Remove mirror_scheduling_tracker and ProjectImporScheduleWorker's needs_own_queue tag
What does this MR do and why?
For #340630 (closed)
In !81249 (merged), we introduced a counter to track the scheduling jobs of ProjectImporScheduleWorker. This counter acts as a replacement for ProjectImportScheduleWorker.queue_size
in UpdateAllMirrorsWorker. The counter usage was hidden behind mirror_scheduling_tracker
. The flag as rolled out successfully a while ago.
This MR cleans up the tag and corresponding code paths. Besides, it also removes needs_own_queue
tag from ProjectImportScheduleWorker. This change set that worker's queue to default
, instead of project_import_schedule
. The shard handling jobs from this worker stays the same - still catchall
.
Screenshots or screen recordings
N/A
How to set up and validate locally
Validate the functionality of UpdateAllMirrorsWorker
- Start one Rails console session (Console A):
loop do
puts "current_scheduling: #{Gitlab::Mirror.current_scheduling}"
puts "queue_size: #{ProjectImportScheduleWorker.queue_size}"
sleep 0.5
end
- Start another Rails console session (Console B). Run the following command to start an UpdateAllMirrorsWorker:
UpdateAllMirrorsWorker.new.perform_async
- Console A prints out the following lines, indicates that
Gitlab::Mirror.current_scheduling
and actual queue size are very close, some seconds apart. Looking at the logs, UpdateAllMirrorsWorker job is blocked until the number drops to 0.
job_tracker: 19
queue_size: 19
job_tracker: 12
queue_size: 10
job_tracker: 4
queue_size: 3
job_tracker: 2
queue_size: 0
job_tracker: 0
queue_size: 0
...
- Look at Sidekiq admin dashboard and logs to see UpdateAllMirrorsWorker is rescheduled
Validate the queue of ProjectImportScheduleWorker
- Step 1: Set the routing_rules configuration in
config/gitlab.yml
to following:
- ["tags=needs_own_queue", null]
- ["worker_name=AuthorizedProjectUpdate::UserRefreshFromReplicaWorker,AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker", "quarantine"] # move this to the quarantine shard
- ["worker_name=RepositoryImportWorker", "imports"] # imports
- ["resource_boundary=cpu&urgency=high", "urgent_cpu_bound"] # urgent-cpu-bound
- ["resource_boundary=memory", "memory_bound"] # memory-bound
- ["feature_category=global_search&urgency=throttled", "elasticsearch"] # elasticsearch
- ["resource_boundary!=cpu&urgency=high", "urgent_other"] # urgent-other
- ["resource_boundary=cpu&urgency=default,low", "low_urgency_cpu_bound"] # low-urgency-cpu-bound
- ["feature_category=database&urgency=throttled", "database_throttled"] # database-throttled
- ["feature_category=gitaly&urgency=throttled", "gitaly_throttled"] # gitaly-throttled
- ["*", "default"] # catchall on k8s
- Step 2: check the queue of ProjectImportScheduleWorker
[1] pry(main)> ProjectImportScheduleWorker.queue
=> "default"
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.