RPC ConsistencyCheck and Praefect "reconcile" subcommand

Part of #2448 (closed)

ConsistencyCheckRPC

This change introduces a new Praefect RPC, /gitaly.PraefectInfoService/ConsistencyCheck, that performs a consistency check on all repos hosted by the requested storage. Additionally, it will schedule a replication job for the target repository if the checksums between it and the reference repository do not match.

Reconcile Subcommand

Additionally, a new Praefect subcommand "reconcile" is available. This subcommand will invoke the the new RPC on a Praefect cluster. The user provides a virtual storage, a target storage, and optionally a reference storage to perform the check. See the logs below (captured from GDK) to see it in action.

Reconcile Usage Examples

When running the subcommand on a fresh new secondary storage:

> praefect -config ./gitaly/praefect.config.toml reconcile -virtual_storage default -target praefect-internal-1 -reference praefect-internal-0
INFO[0000] parsed scheme: "unix"                         pid=10196 system=system
INFO[0000] scheme "unix" not registered, fallback to default scheme  pid=10196 system=system
INFO[0000] ccResolverWrapper: sending update to cc: {[{unix:///Users/paulokstad/gitlab-development-kit/praefect.socket 0  <nil>}] <nil>}  pid=10196 system=system
INFO[0000] ClientConn switching balancer to "pick_first"  pid=10196 system=system
INFO[0000] pickfirstBalancer: HandleSubConnStateChange: 0xc000298620, CONNECTING  pid=10196 system=system
INFO[0000] CPU time info is unavailable on non-linux or appengine environment.  pid=10196 system=system
INFO[0000] pickfirstBalancer: HandleSubConnStateChange: 0xc000298620, READY  pid=10196 system=system
2020/03/14 09:29:42 INCONSISTENT: @hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.wiki.git - replication scheduled: #1
2020/03/14 09:29:42 INCONSISTENT: @hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.git - replication scheduled: #2
2020/03/14 09:29:42 INCONSISTENT: @hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki.git - replication scheduled: #3
2020/03/14 09:29:42 INCONSISTENT: @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.wiki.git - replication scheduled: #4
2020/03/14 09:29:42 INCONSISTENT: @hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.wiki.git - replication scheduled: #5
2020/03/14 09:29:42 INCONSISTENT: @hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.wiki.git - replication scheduled: #6
2020/03/14 09:29:42 INCONSISTENT: @hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git - replication scheduled: #7
2020/03/14 09:29:42 INCONSISTENT: @hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki.git - replication scheduled: #8
2020/03/14 09:29:42 INCONSISTENT: @hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git - replication scheduled: #9
2020/03/14 09:29:42 INCONSISTENT: @hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki.git - replication scheduled: #10
2020/03/14 09:29:42 INCONSISTENT: @hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.wiki.git - replication scheduled: #11
2020/03/14 09:29:42 INCONSISTENT: @hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.wiki.git - replication scheduled: #12
2020/03/14 09:29:42 INCONSISTENT: @hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918.wiki.git - replication scheduled: #13
2020/03/14 09:29:42 INCONSISTENT: @hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git - replication scheduled: #14
2020/03/14 09:29:42 INCONSISTENT: @hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git - replication scheduled: #15
2020/03/14 09:29:42 INCONSISTENT: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451.git - replication scheduled: #16
2020/03/14 09:29:42 INCONSISTENT: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451.wiki.git - replication scheduled: #17
2020/03/14 09:29:42 INCONSISTENT: @hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61.wiki.git - replication scheduled: #18
2020/03/14 09:29:42 INCONSISTENT: @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git - replication scheduled: #19
2020/03/14 09:29:42 INCONSISTENT: @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.wiki.git - replication scheduled: #20
2020/03/14 09:29:42 INCONSISTENT: @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.wiki.git - replication scheduled: #21
2020/03/14 09:29:42 INCONSISTENT: @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git - replication scheduled: #22
2020/03/14 09:29:42 INCONSISTENT: @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git - replication scheduled: #23
2020/03/14 09:29:42 INCONSISTENT: @hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb.wiki.git - replication scheduled: #24
2020/03/14 09:29:42 INCONSISTENT: @hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git - replication scheduled: #25
2020/03/14 09:29:42 INCONSISTENT: @hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.wiki.git - replication scheduled: #26
2020/03/14 09:29:42 INCONSISTENT: @hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git - replication scheduled: #27
2020/03/14 09:29:42 INCONSISTENT: @hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki.git - replication scheduled: #28

... when the node has reconciled, no output will be returned on subsequent runs.