Spawn dedicated metrics-server process on puma-master
This is a follow-up to #350546 (closed).
We currently run an in-process HTTP server on puma_master
to serve metrics emitted from workers into Prometheus. We are looking to extract this server into its own process for reasons of efficiency and availability.
In this issue we are looking to spawning WebExporter
(the metrics server) in a separate process instead.
This also requires us to introduce some sort of control-loop in this process to restart that server should it go away, as we do with sidekiq-cluster.
Rollout
-
Introduce the server process: !82478 (merged) -
Enable the env var on staging: gitlab-com/gl-infra/k8s-workloads/gitlab-com!1578 (merged) -
Verify change on staging: metrics should still be served for Puma. -
Enable the env var for production canaries -
Verify the change: -
Application metrics are served, e.g. puma_*
metrics -
Process memory use stable -
Container memory use stable
-
-
Enable the env var everywhere -
Verify the change -
Clean up the environment variable
Edited by Matthias Käppler