Reconciliation should report progress and warn user

Closes #2621 (closed)

This change is a step towards improving the progress reporting of reconciliation while also providing the user more information about what will change. The command behavior has changed to perform a dry run (side effect free) by default. To perform actual repairs, the option -live-run must be provided. The log outputs are much more verbose, as shown below:

Dry runs

One consistent repo:

~/gitlab-development-kit > gitaly/_build/bin/praefect -config gitaly/praefect.config.toml reconcile -virtual default -target praefect-internal-1
2020/05/07 09:17:24 Performing a DRY run - no changes will be made until '-live-run' flag is provided
2020/05/07 09:17:24 Checking consistency...
2020/05/07 09:17:24 Reference storage being used: praefect-internal-0
2020/05/07 09:17:24 CONSISTENT: @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git
2020/05/07 09:17:24 FINISHED: 1 repos were checked for consistency

One inconsistent repo:

~/gitlab-development-kit > rm -rf repositories/praefect-internal-1/@hashed 
~/gitlab-development-kit > gitaly/_build/bin/praefect -config gitaly/praefect.config.toml reconcile -virtual default -target praefect-internal-1
2020/05/07 09:20:26 Performing a DRY run - no changes will be made until '-live-run' flag is provided
2020/05/07 09:20:26 Checking consistency...
2020/05/07 09:20:26 Reference storage being used: praefect-internal-0
2020/05/07 09:20:26 INCONSISTENT: Repo @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git has checksum null on target but checksum 0000000000000000000000000000000000000000 on reference storage praefect-internal-0
2020/05/07 09:20:26 FINISHED: 1 repos were checked for consistency

Live Runs

With inconsistent repo:

~/gitlab-development-kit > gitaly/_build/bin/praefect -config gitaly/praefect.config.toml reconcile -virtual default -target praefect-internal-1 -live-run
2020/05/07 09:34:35 Performing a LIVE run - any detected inconsistencies on reference will be overwritten on target
2020/05/07 09:34:35 Checking consistency...
2020/05/07 09:34:35 Reference storage being used: praefect-internal-0
2020/05/07 09:34:35 INCONSISTENT: Repo @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git has checksum null on target but checksum 0000000000000000000000000000000000000000 on reference storage praefect-internal-0
2020/05/07 09:34:35 SCHEDULED: Replication job 2 will update repo @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git
2020/05/07 09:34:35 FINISHED: 1 repos were checked for consistency

With repaired (consistent) repo:

~/gitlab-development-kit > gitaly/_build/bin/praefect -config gitaly/praefect.config.toml reconcile -virtual default -target praefect-internal-1 -live-run
2020/05/07 09:35:56 Performing a LIVE run - any detected inconsistencies on reference will be overwritten on target
2020/05/07 09:35:56 Checking consistency...
2020/05/07 09:35:56 Reference storage being used: praefect-internal-0
2020/05/07 09:35:56 CONSISTENT: @hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git
2020/05/07 09:35:56 FINISHED: 1 repos were checked for consistency