Deprecation: Sharing metrics + health-checks port for Sidekiq
Deprecation Summary
Prior to GitLab 14.7, Sidekiq offered HTTP endpoints for two different tasks, that were served by a single in-process HTTP server:
-
/metrics
to serve a Prometheus scraper, exporting metrics on behalf of all Sidekiq worker processes started bysidekiq-cluster
. -
/readiness
and/liveness
to serve health-checks; these were initially introduced to serve thekubelet
in a Kubernetes based setup, such as when deploying a Cloud Native GitLab (CNG) through our Helm Charts, however, any other HTTP based observability solution could potentially be consuming these.
With &6409 (closed), those functionalities have been segregated such that:
-
/metrics
are now served by a dedicated server process, not an individual worker. -
/readiness
and/liveness
continue to be served by an in-worker HTTP server.
With two separate servers running on the same host or pod, we need to allocate two different ports for clients to reach them, and make sure that those clients (kubelet probes, prometheus scrapers, and any other clients) are configured to target those ports accordingly.
For backwards-compatibility we currently default health-checks to be exported from the same port as metrics, and do not start the server process until we detect that two separate ports are actually configured.
Going forward, we require all GitLab admins to either expect the new defaults, or explicitly configure separate ports for metrics and health-checks. To that end, we introduced a new set of settings for Sidekiq Monitoring: the health_checks
key.
Affected Topology
- SaaS: not affected; we will handle this transparently via our Chart overrides for SaaS.
-
Self-managed:
-
Omnibus and from-source: Affected only if they were monitoring Sidekiq workers via
/readiness|/liveness
through custom setup/configuration. Those integrations will need to choose a new port. If not, then we will use a different default port going forward and the change should be transparent, unless the default port we choose is allocated already, in which case a custom port needs to be set (we cannot know this upfront). -
Chart users: Affected only if they are overriding default Chart settings and are consuming
/readiness|/liveness
in a way that expects the same port as/metrics
. If not, we will choose a new default port for these health-checks.
-
Omnibus and from-source: Affected only if they were monitoring Sidekiq workers via
Affected Tier
- Free
- Premium
- Ultimate
Checklist
-
@mention
your stage's stable counterparts on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager.- To see who the stable counterparts are for a product team visit product categories
- If there is no stable counterpart listed for Sales/CS please mention
@timtams
- If there is no stable counterpart listed for Support please
@mention
@gitlab-com/support/managers
- If there is no stable counterpart listed for Marketing please mention
@williamchia
- If there is no stable counterpart listed for Sales/CS please mention
- To see who the stable counterparts are for a product team visit product categories
-
@mention
your GPM so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change.
Deprecation Milestone
Planned Removal Milestone
%15.0 (this is a breaking change and the removal will happen on the next major release)
Links
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
- https://docs.gitlab.com/charts/charts/gitlab/sidekiq
- https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes
- https://about.gitlab.com/handbook/product/gitlab-the-product/#process-for-deprecating-and-removing-a-feature
- https://docs.gitlab.com/ee/update/deprecations
Related Tasks
-
@rzwambag Add documentation on how to configure health-check
settings (#345803 (closed)) -
@rzwambag Coordinate with groupdistribution on whether we need to add a note in deprecations.rb (documentation). If needed, also update deprecations.rb
-
@iroussos add deprecation warning - MR: !77962 (merged) Related docs: