Geo: Better handle SSH/git failures
My Geo primary/secondary have something out of sync, so I see this:
2017-03-22_23:02:13.50289 2017-03-22T23:02:13.502Z 1493 TID-ot9xi75ak GeoRepositoryFetchWorker JID-998fae4ad9c4dd03e55c3a59 INFO: start
2017-03-22_23:02:26.10321 2017-03-22T23:02:26.102Z 1493 TID-ot9x2wxa8 GeoRepositoryFetchWorker JID-5e0d5a453c115f4c7aef7e4b INFO: fail: 36.903 sec
2017-03-22_23:02:26.10323 2017-03-22T23:02:26.102Z 1493 TID-ot9x2wxa8 WARN: {"context":"Job raised exception","job":{"class":"GeoRepositoryFetchWorker","args":[5,"git@stanhu-geo-primary.gitlap.com:root/new-project2.git"],"retry":3,"queue":"geo_repository_update","jid":"5e0d5a453c115f4c7aef7e4b","created_at":1490161193.7184846,"enqueued_at":1490161193.7186701,"error_message":"error: cannot lock ref 'refs/heads/master': is at 841d8f704001f56aa5eb140848835637a6e7d8d7 but expected 7509a3defe10deb58fe4151a973132fc85c33fdf\nFrom stanhu-geo-primary:root/new-project2\n ! 7509a3d..841d8f7 master -> master (unable to update local ref)\n","error_class":"Gitlab::Shell::Error","failed_at":1490223746.1023812,"retry_count":0},"jobstr":"{\"class\":\"GeoRepositoryFetchWorker\",\"args\":[5,\"git@stanhu-geo-primary.gitlap.com:root/new-project2.git\"],\"retry\":3,\"queue\":\"geo_repository_update\",\"jid\":\"5e0d5a453c115f4c7aef7e4b\",\"created_at\":1490161193.7184846,\"enqueued_at\":1490161193.7186701}"}
2017-03-22_23:02:26.10324 2017-03-22T23:02:26.102Z 1493 TID-ot9x2wxa8 WARN: Gitlab::Shell::Error: error: cannot lock ref 'refs/heads/master': is at 841d8f704001f56aa5eb140848835637a6e7d8d7 but expected 7509a3defe10deb58fe4151a973132fc85c33fdf
2017-03-22_23:02:26.10324 From stanhu-geo-primar:root/new-project2
2017-03-22_23:02:26.10324 ! 7509a3d..841d8f7 master -> master (unable to update local ref)
This fails badly and retries over and over.
We should also improve the health status to report these failures.
/cc: @dbalexandre