Add Sidekiq execution SLI as apdex
What does this MR do and why?
This MR adds Sidekiq execution metrics as apdex and error rate counters, which will then be aggregated as application SLIs as described in https://docs.gitlab.com/ee/development/application_slis/.
Resolves Add Sidekiq execution Application SLIs to the R... (gitlab-com/gl-infra/scalability#1638 - closed) (check the issue for detailed requirement of the metrics)
How to set up and validate locally
-
Run
gdk
-
Check that
gitlab_sli_sidekiq_execution_*
metrics are present:❯ curl 'gdk.test:3807/metrics' -s | grep gitlab_sli_sidekiq_execution_ ...
-
Spawn a
gdk rails c
console -
Push a job to any worker, eg
Chaos::SleepWorker.perform_async(1)
-
The counters will get incremented:
❯ curl 'gdk.test:3807/metrics' -s | grep gitlab_sli | grep SleepWorker gitlab_sli_sidekiq_execution_apdex_success_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 1 gitlab_sli_sidekiq_execution_apdex_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 1 gitlab_sli_sidekiq_execution_error_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 0 gitlab_sli_sidekiq_execution_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 1
-
To emulate an error, run
Chaos::SleepWorker.perform_async(1, 2, 3)
-
The error counter will get incremented:
❯ curl 'gdk.test:3807/metrics' -s | grep gitlab_sli | grep SleepWorker gitlab_sli_sidekiq_execution_apdex_success_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 1 gitlab_sli_sidekiq_execution_apdex_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 1 gitlab_sli_sidekiq_execution_error_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 1 gitlab_sli_sidekiq_execution_total{feature_category="not_owned",urgency="low",worker="Chaos::SleepWorker"} 2
-
To test apdex failure, run
Chaos::SleepWorker.perform_async(301)
. This will exceed the requirement of 300s. -
gitlab_sli_sidekiq_execution_apdex_total
will get incremented withoutgitlab_sli_sidekiq_execution_apdex_success_total
incremented.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.