Skip to content

feat: Add monitoring metrics for ApplicationRateLimiter

Milad Irannejad requested to merge miladx/add-app-rate-limiter-metrics into master

What does this MR do and why?

This MR adds three new OpenMetrics/Prometheus metrics for GitLab custom-made ApplicationRateLimiter. ApplicationRateLimiter, similar to RackAttack, enforces a set of predefined throttling rules. Unlike RackAttack which operates as a middleware, ApplicationRateLimiter can be used at the Controller or API level.

This change will enable us to build a better dashboard to troubleshoot rate-limiting and throttling issues. Please see the following issues for more context:

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

TBD

How to set up and validate locally

  1. Apply this diff
diff --git a/app/workers/chaos/sleep_worker.rb b/app/workers/chaos/sleep_worker.rb
index 149bab5d9d3c..b60d819d628d 100644
--- a/app/workers/chaos/sleep_worker.rb
+++ b/app/workers/chaos/sleep_worker.rb
@@ -10,6 +10,8 @@ class SleepWorker # rubocop:disable Scalability/IdempotentWorker
     include ChaosQueue

     def perform(duration_s)
+      ::Gitlab::ApplicationRateLimiter.throttled?(:user_following, scope: [])
+
       Gitlab::Chaos.sleep(duration_s)
     end
   end
  1. Enable monitoring.sidekiq_exporter.enabled: true
  2. Run gdk restart rails
  3. Open gdk rails console and run:
30.times { Chaos::SleepWorker.perform_async(1) }
  1. Run curl localhost:3807/metrics | rg gitlab_application 
➜  gitlab git:(miladx/add-app-rate-limiter-metrics) ✗ curl   localhost:3807/metrics | rg gitlab_application
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP gitlab_application_rate_limiter_throttles_total Multiprocess metric
# TYPE gitlab_application_rate_limiter_throttles_total histogram
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="+Inf",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="0.25",peek="false",queue="default",throttle_key="user_following"} 35
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="0.5",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="0.75",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_bucket{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",le="1",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_count{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",peek="false",queue="default",throttle_key="user_following"} 40
gitlab_application_rate_limiter_throttles_total_sum{endpoint_id="Chaos::SleepWorker",feature_category="not_owned",peek="false",queue="default",throttle_key="user_following"} 5.2
Edited by Sylvester Chin

Merge request reports