Skip to content

gitaly: Tune StatefulSet probes

Emily Chui requested to merge echui-tune-gitaly-readiness-liveness-probes into master

What does this MR do?

Gitaly is expected to boot in less than a second or two, at which point it's fully available to serve requests. Startup time is crucial for the service due to the implementation of transactions and a write-ahead log, due to roll out in the near future. When these components are enabled, it will no longer possible to have multiple instances of Gitaly operating against the same storage.

Such a situation occurs when Gitaly is being upgraded to a new version. In a VM deployment, Gitaly would utilise the tableflip library which essentially swaps the underlying binary while draining existing requests from the old process. In Kubernetes, this will be a pod rotation.

Work being completed in gitlab-org/gitaly#6209 (closed) is attempting to "bridge the gap" that opens during a Gitaly upgrade that cannot utilise the tableflip method. We will have to rely on client retry logic to minimise disruptions during the upgrade.

Tune the various probes for Gitaly's StatefulSet deployment. We want to probe the healthcheck as soon as practicable to minimise the time taken for Kubernetes to route traffic to a fresh Gitaly pod.

To further enable the lifecycle efficiency, update startupProbe to be on by default.

Related issues

Relates to #5636 (closed)

Author checklist

For general guidance, please follow our Contributing guide.

Required

For anything in this list which will not be completed, please provide a reason in the MR discussion.

  • Merge Request Title and Description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • MR has a green pipeline.
  • Documentation created/updated.
  • Tests added/updated, and test plan for scenarios not covered by automated tests.
  • Equivalent MR/issue for omnibus-gitlab opened.

Reviewers checklist

Edited by Jon Doveston

Merge request reports