Pause gitaly on upgrades when the operator is enabled
Summary
Gitaly should be paused during the helm upgrade, similar to how unicorn and sidekiq are.
- Add a pause job for the gitaly statefulset, similar to the pre-hook jobs we have for unicorn and sidekiq
Original description
We see in weekly demos that when upgrading to a new version using operator, Gitaly gets rolled over as soon as operator starts its job. While that is happening, operator moves to pre-migrations stage (this may be a bug, being tracked in https://gitlab.com/charts/gitlab/issues/1261). After pre-migrations job, we roll over the services which require this new data - unicorn, sidekiq and gitaly.
However, it is not clear if we need to actually rollover gitaly again after pre-migrations when it was already rolled over earlier and have the latest code.
@twk3 mentioned takeoff also did something similar while deploying to GitLab.com, so there may be some underlying reason for this. We need to find out and either document or change it as necessary.