Fix PauseControl sidekiq middleware
What does this MR do and why?
During the Zoekt incident gitlab-com/gl-infra/production#17266 (closed) we noticed that PauseControl Sidekiq middleware didn't pause all Zoekt indexing operations as we expected. After some debugging I've noticed that the client part of the middeware receives worker_class
, but the server one receives worker instance, which caused this bug for the latter.
Because of this mismatch the second part wasn't working properly and was also returning strategy: :none
.
The simplest way to test it is
Gitlab::SidekiqMiddleware::PauseControl::WorkersMap.strategy_for(worker: Zoekt::IndexerWorker.new)
On master it's:
[1] pry(main)> Gitlab::SidekiqMiddleware::PauseControl::WorkersMap.strategy_for(worker: Zoekt::IndexerWorker.new)
=> nil
[2] pry(main)> Gitlab::SidekiqMiddleware::PauseControl::WorkersMap.strategy_for(worker: Zoekt::IndexerWorker)
=> :zoekt
And on this branch:
[1] pry(main)> Gitlab::SidekiqMiddleware::PauseControl::WorkersMap.strategy_for(worker: Zoekt::IndexerWorker.new)
=> :zoekt
[2] pry(main)> Gitlab::SidekiqMiddleware::PauseControl::WorkersMap.strategy_for(worker: Zoekt::IndexerWorker)
=> :zoekt
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Set up Zoekt
- Check out master
git checkout master
- Restart Sidekiq
gdk restart rails-background-jobs
- Truncate zoekt index
Gitlab::Search::Zoekt::Client.instance.truncate
- Ensure that you don't have files in
$GDK_DIR/zoekt-data/development/index
- Restart rails console and perform these commands (in that order). This is specifically to schedule the job in the future before we pause indexing
Zoekt::IndexerWorker.perform_in(65, 7, { "foo" => Time.now.to_i }) ::Feature.enable(:zoekt_pause_indexing)
- Monitor
$GDK_DIR/zoekt-data/development/index
andtail -f log/sidekiq.log | fgrep Zoekt
- In 65 seconds you should see that the job has been executed and index files in
$GDK_DIR/zoekt-data/development/index
appeared - Checkout this branch
fix-pause-control-middleware
- Restart Sidekiq
gdk restart rails-background-jobs
- Truncate zoekt index, disable the FF, and clear the ZSET
::Feature.disable(:zoekt_pause_indexing) PauseControl::ResumeWorker.new.perform # Wait a few seconds so that existing jobs complete Gitlab::Search::Zoekt::Client.instance.truncate
- Ensure that you don't have files in
$GDK_DIR/zoekt-data/development/index
- Perform these commands (in that order). This is specifically to schedule the job in the future before we pause indexing
Zoekt::IndexerWorker.perform_in(65, 7, { "foo" => Time.now.to_i }) ::Feature.enable(:zoekt_pause_indexing)
- Monitor
$GDK_DIR/zoekt-data/development/index
andtail -f log/sidekiq.log | fgrep Zoekt
- Wait for log records with
"job_status":"paused"
- In 65 seconds you should not see that the job has been executed and the
$GDK_DIR/zoekt-data/development/index
directory should be empty - You can also check that this commands returns
1
Gitlab::SidekiqMiddleware::PauseControl::PauseControlService.queue_size('Zoekt::IndexerWorker')
- Do not forget to unpause indexing
::Feature.disable(:zoekt_pause_indexing)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.