Add rate limiter for PullRequest importer
What does this MR do and why?
- Contributes to #345922 (closed)
- Feature flag: #353217 (closed)
Problem
We don't have a limitation for a number of PullRequest importer jobs. The parallel importer process can generate thousands of them. It puts pressure on other services.
Solution
Limit the number of created jobs by spreading the load for a larger period of time.
New settings max_concurrency
and backoff
allow to setup the
desired size and delay between batches.
The importer was causing problems when we had ~1000 jobs per second. I chose 200 concurrent jobs as a number that is relatively high but well below the critical threshold. It can be adjusted later if necessary.
Screenshots or screen recordings
Sidekiq::ScheduledSet.new.select { |a| a['class'] == 'Gitlab::GithubImport::ImportPullRequestWorker' }.map { |a| a['scheduled_at'] }.map { |time| Time.at(time).to_datetime.to_s }
=> ["2022-02-21T19:40:03+01:00", "2022-02-21T19:39:03+01:00", "2022-02-21T19:38:03+01:00", "2022-02-21T19:37:03+01:00", "2022-02-21T19:36:03+01:00"]
How to set up and validate locally
- Enable feature flag
Feature.enable(:limit_parallel_import)
-
Set
max_concurrency
to 1 (to simplify the testing) -
Restart sidekiq (
gdk restart rails-background-jobs
) -
Open new project page (http://127.0.0.1:3000/projects/new)
-
Choose
Import project
->GitHub
-
Add a personal token if not set
-
Select and start importing a project with open Pull Requests
-
In Rails console execute (need to retry several times)
Sidekiq::ScheduledSet.new.select { |a| a['class'] == 'Gitlab::GithubImport::ImportPullRequestWorker' }.map { |job| Time.at(job['scheduled_at']).to_datetime.to_s }
=> ["2022-02-21T19:40:03+01:00", "2022-02-21T19:39:03+01:00", "2022-02-21T19:38:03+01:00", "2022-02-21T19:37:03+01:00", "2022-02-21T19:36:03+01:00"]
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.