Draft: PDM to migrate queued and future Sidekiq jobs
MR merged together with !142577 (merged) for easier merge coordination.
What does this MR do and why?
Part of gitlab-com/gl-infra/scalability#2720 (closed)
Migrate jobs that don't belong to any queue from the routing rules to their correct queue.
Why is the migration needed?
- Before %16.0, jobs are pushed to their worker-named queue by default, equivalent to
[['*', null]]
in routing rules terms. - On %16.0, routing rules are defaulted to
[['*', 'default']]
which means all jobs from all worker class are pushed to thedefault
queue. However, Sidekiq server still listens to all queues by default (iesidekiq['queue_groups'] = ['*']
). - In !142577 (merged) %16.9, we're changing queue_groups to be
default,mailers
if routing rules aren't set. - Scheduled and retried jobs might still be left in their worker queues. For example, scheduled/retried jobs while in pre-16.0 could be scheduled in the distant future, and these will remain in their individual queues. This migration ensures no jobs will be lost while upgrading to %16.9
Note:
- The code is mostly copy-pasted from https://gitlab.com/gitlab-org/gitlab/-/blob/7066e106511295f69e380eef89c7e9a805a29330/lib/gitlab/sidekiq_migrate_jobs.rb which is a Rake task designed to migrate the jobs whenever routing rules are updated.
- We have tried to introduce the PDM before, but was reverted/cancelled because different approach was taken back then: a. Queued jobs: !100392 (merged) b. Future jobs: !103001 (closed)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Edited by Gregorius Marco