Reduce time for service discovery to update after database replica dns change
During the postgres 14 upgrade of the main database on GitLab.com (gitlab-com/gl-infra&642 (closed)), it took roughly 40 minutes after replicas were set to point to the pg14 cluster in consul before database traffic began moving from the pg12 replicas to the pg14 replicas.
Identify why this change took so long, and reduce it significantly for future maintenance.
During this upgrade, first the patroni-main-103 replica was removed from load balancing, and quickly traffic to it stopped. Then the other replicas were removed, and it took the above mentioned 40 minutes for traffic to them to stop.