Draft: Transit to non-namespaced keys in Sidekiq
What does this MR do and why?
Refer to gitlab-com/gl-infra/scalability#1982 (closed) for motivations and discussions
This MR adds several patches which feature-flag-gated logic to switch from resque:gitlab
namespace to non-namespaced keys in sidekiq.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
- Create dummy worker
In a file named
app/workers/fake_worker.rb
, create.
# frozen_string_literal: true
class FakeWorker
include ApplicationWorker
data_consistency :always
idempotent!
def perform(duration, delay = nil)
if delay.nil?
FakeWorker.perform_async(duration)
else
FakeWorker.perform_at(delay, duration, delay)
end
sleep(duration)
end
end
- Restart gdk
gdk restart
- Observe states pre-feature-flag toggle by starting up a redis-cli session
gdk redis-cli -n 1
> keys * // should see all keys in resque:gitlab namespace
- Trigger dummy jobs
gdk rails c
[1] pry(main)> (1..20).each {|n| FakeWorker.perform_async(n)}
[2] pry(main)> (1..20).each {|n| FakeWorker.perform_async(n)} // create duplicates
[3] pry(main)> (1..20).each {|n| FakeWorker.perform_at(n, n, n)}
Check llen resque:gitlab:queue:default
and llen <working queue>
. The name of the working queue can be retrieved via keys *working*
.
- Toggle feature flag
# within gdk rails c
Feature.enable(:disable_sidekiq_namespace, true)
- Observe migration of jobs
- Queues: Compare
llen resque:gitlab:queue:default
vsllen queue:default
, the former should reduce to 0 while the latter start populating. - ScheduleSet: Compare
zcard resque:gitlab:schedule
vszcard schedule
, the former should stop changing while the latter is populatled.
Other keys to observe
-
processes
should contain the new process -
cron_jobs
and respectivecron_jobs:*
should be migrated
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Sylvester Chin