Select worker to schedule BG migrations
What does this MR do and why?
Related to #343055 (closed)
Prepares for the addition of a separate BG migration worker for the ci database by conditionally scheduling jobs based on the target database of the migration. The target database for a BG migration is the "primary" database for the migration, where tracking data for that migration will live.
While a migration job is unrestricted to use connections to either database, a developer adding a BG migration will have to choose the database/queue where the migration information will be tracked and executed. For most BG migrations that only target tables in a single database, the BG migration will be queued to also use the tracking tables in the same database.
For now we only support scheduling to the main database, but we will add the working and scheduling for the ci database in a future change.
How to set up and validate locally
- Create a test background migration in the rails console:
module Gitlab module BackgroundMigration class MyFancyMigration def perform(start_id, end_id) puts "fancy: #{start_id}, #{end_id}" end end end end
- Create and run a migration to enqueue the jobs for a specific table:
migration = Class.new(Gitlab::Database::Migration[1.0]) do def up queue_background_migration_jobs_by_range_at_intervals(define_batchable_model(:issues), 'MyFancyMigration', 2.minutes, batch_size: 2, initial_delay: 1.week, track_jobs: true) end end migration.new.up # -- Scheduled 39 MyFancyMigration jobs with a maximum of 2 records per batch and an interval of 120 seconds.
- Verify the scheduled set matches the number of jobs reported above:
Sidekiq::ScheduledSet.new.select { |j| j.klass == 'BackgroundMigrationWorker' && j.args.first == 'MyFancyMigration' }.size # 39
- Create and run a migration to finalize and force the jobs to run:
migration2 = Class.new(Gitlab::Database::Migration[1.0]) do def up finalize_background_migration('MyFancyMigration') end end migration2.new.up # you should see the job output as they run
- Verify the queue is empty, and the migrations are complete:
Sidekiq::ScheduledSet.new.select { |j| j.klass == 'BackgroundMigrationWorker' && j.args.first == 'MyFancyMigration' }.size # 0
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.