Option for failover to put out-of-date repositories in read-only mode

Problem to solve

Praefect has a failover mechanism, where if it detects that a Gitaly node is unresponsive, it will promote a secondary storage server to primary. If this secondary server could be missing data still from the old primary. This in itself would allow Gitaly to serve stale data. However, after serveral mutations on the repository, it could end up in a state where data-loss occurs. As such, the current implementation is great for demo's, or instances where velocity is more important than data integrity. The current approach will not work for gitlab.com

Solution

In we're working on reporting of potential data loss, this issue proposes the next iteration of it; putting each of those repositories in a read only state, until manual intervention from an admin to mark it operational again. Allowing all administrators to use the current failover mechanism, and resolve data integrety issues manually.

Additional benefits

When moving a repository on a shard, it's put in read only mode for a brief period of time. This is a loosely defined contract right now. Having a praefect managed read only mode for repositories put stronger data integrity.

This approach could remove the need for #2601