Primary Gitaly node specified per repository

Problem to solve

Currently, primary Gitaly nodes are set per virtual storage, so all the repositories on that cluster use the same node as the primary. This can cause issues around resource consumption, scalability, and failover scenarios. The primary node acts as a bottleneck for every repository, so if one repository becomes extremely active, it could impact performance for all other repositories on that cluster.

In order to allow for more control over Gitaly cluster storage, the primary node should be available to be set per repository. This lets admins reduce load on a single node by scaling the primaries out horizontally onto multiple nodes across the cluster. Some problems that this feature will solve are:

• Failovers won't switch most of the repositories in to read-only mode as we can choose the best primary for each repository.
• Write load can be spread across the cluster and a single primary won't be the bottleneck.

Further details

Looking forward, this sets us up to add more functionality around automatic rebalancing, horizontal scaling (by setting the replication factor of each repository), and repository

• Since primary is per repository, this opens up the possibility of no Gitaly node needing every repository on the disk later. In this iteration, each repository would still be replicated on every node.
• Sets a good base for automatic rebalancing in the future, as we can move write heavy repositories to Gitaly nodes that have excess IO capacity.
• Additionally for #3015 (closed), this provides a good base for draining a node in the future to take it down gracefully. One can switch the primary per repository as possible, not only when the whole virtual storage is replicated to the desired new primary.

Proposal

Praefect should pick a random, healthy Gitaly node as a primary for each repository.