DB load balancing service discovery returns incomplete list with Consul
Quoting Consul docs:
When the DNS query is performed using UDP, Consul will truncate the results without setting the truncate bit. This is to prevent a redundant lookup over TCP that generates additional load. If the lookup is done over TCP, the results are not truncated.
We turned service discovery in staging and we found that we're only utilizing 3 replicas out of 5. As a test, we patched Resolv::DNS
to use TCP requester instead of a UDP one, which resulted in the full set being returned.
cc/ @yorickpeterse