Generic BG migration jobs use the correct context
What does this MR do and why?
Related issue: #345624 (closed)
We're currently updating background migrations to work with multiple databases by adding separate workers and queues targeting the main
and ci
databases. Jobs are intended to have the flexibility to connect to the database(s) they need, since some migrations may be generic or even cross-database. For this to work, generic jobs need to have their execution context accessible, so they can act on the appropriate database.
This MR introduces a base class to be used for background migrations. If the job class inherits from this base class, it will be initialized with the base model for the tracking database. The base model can be used either to create other models that target the tracking database, or to retrieve the underlying connection.
We only have one generic migration class that currently needs these changes, but the base class will be expanded in the future to expose other common functionality to migration jobs. Additionally, single-use migration jobs can still benefit from "knowing" their tracking database, since it requires less hardcoded database access.
How to set up and validate locally
To test we'll simulate a partition data migration.
- Create a fake partitioned table:
create table projects_part_2c7d399fd8 (like projects, primary key (id, created_at));
- Create and run a temporary migration to run the backfill:
migration_class = Class.new(Gitlab::Database::Migration[1.0]) do include Gitlab::Database::PartitioningMigrationHelpers def up enqueue_partitioning_data_migration :projects end def assert_table_is_allowed(table_name) # nothing end end migration_class.new.up
- Create and run a temporary migration to finish the backfill:
migration_class = Class.new(Gitlab::Database::Migration[1.0]) do include Gitlab::Database::PartitioningMigrationHelpers def up finalize_backfilling_partitioned_table :projects end def assert_table_is_allowed(table_name) # nothing end end migration_class.new.up
- Verify the migration has completed:
select count(*) from projects where id not in (select id from projects_part_2c7d399fd8); -- should be 0 select status, count(*) from background_migration_jobs where class_name = 'Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable' and arguments->>2 = 'projects' group by 1; -- they should all have status of 1
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.