feat: Add monitoring metrics for ApplicationRateLimiter
What does this MR do and why?
This MR adds three new OpenMetrics/Prometheus metrics for GitLab custom-made ApplicationRateLimiter. ApplicationRateLimiter, similar to RackAttack, enforces a set of predefined throttling rules. Unlike RackAttack which operates as a middleware, ApplicationRateLimiter can be used at the Controller or API level.
This change will enable us to build a better dashboard to troubleshoot rate-limiting and throttling issues. Please see the following issues for more context:
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
TBD
How to set up and validate locally
- Apply this diff
diff --git a/app/workers/chaos/sleep_worker.rb b/app/workers/chaos/sleep_worker.rb
index 149bab5d9d3c..b60d819d628d 100644
--- a/app/workers/chaos/sleep_worker.rb
+++ b/app/workers/chaos/sleep_worker.rb
@@ -10,6 +10,8 @@ class SleepWorker # rubocop:disable Scalability/IdempotentWorker
include ChaosQueue
def perform(duration_s)
+ ::Gitlab::ApplicationRateLimiter.throttled?(:user_following, scope: [])
+
Gitlab::Chaos.sleep(duration_s)
end
end
- Enable
monitoring.sidekiq_exporter.enabled: true
- Run
gdk restart rails
- Open
gdk rails console
and run:
30.times { Chaos::SleepWorker.perform_async(1) }
- Run
curl localhost:3807/metrics | rg gitlab_application
➜ gitlab git:(miladx/add-app-rate-limiter-metrics) ✗ curl localhost:3807/metrics | rg gitlab_application
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP gitlab_application_rate_limiter_throttles_total Multiprocess metric
# TYPE gitlab_application_rate_limiter_throttles_total histogram
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="+Inf",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="0.25",peek="false",queue="default",throttle_key="user_following"} 35
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="0.5",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="0.75",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="1",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_count{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_sum{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",peek="false",queue="default",throttle_key="user_following"} 5.2
Edited by Sylvester Chin