Separate local health checking from the consensus in HealthManager
Currently, Praefect's nodes.HealthManager
only returns the consensus of Gitaly nodes' health statuses. This is sufficient for components like the reconciler, as they don't make any requests to the nodes directly, only schedule the jobs. Likewise, the primary elector wants to elect a primary that is healthy by the majority of Praefect's. Some components, like the coordinator, need to know the healthiness of the local connection to avoid performing requests to unhealthy nodes. Coordinator performs the requests in sync, so calling unhealthy nodes should be avoided.
Given the repository specific primaries are not elected, we could completely avoid loading the health consensus from the database and simply get it in each query that needs it directly. If this is done, we should likely include a stored procedure to do so to avoid duplicating the code for determining the healthy nodes.
Related to #3133 (closed)