Skip to content

Generic BG migration jobs use the correct context

Patrick Bair requested to merge pb-provide-context-for-bg-migrations into master

What does this MR do and why?

Related issue: #345624 (closed)

We're currently updating background migrations to work with multiple databases by adding separate workers and queues targeting the main and ci databases. Jobs are intended to have the flexibility to connect to the database(s) they need, since some migrations may be generic or even cross-database. For this to work, generic jobs need to have their execution context accessible, so they can act on the appropriate database.

This MR introduces a base class to be used for background migrations. If the job class inherits from this base class, it will be initialized with the base model for the tracking database. The base model can be used either to create other models that target the tracking database, or to retrieve the underlying connection.

We only have one generic migration class that currently needs these changes, but the base class will be expanded in the future to expose other common functionality to migration jobs. Additionally, single-use migration jobs can still benefit from "knowing" their tracking database, since it requires less hardcoded database access.

How to set up and validate locally

To test we'll simulate a partition data migration.

  1. Create a fake partitioned table:
    create table projects_part_2c7d399fd8 (like projects, primary key (id, created_at));
  2. Create and run a temporary migration to run the backfill:
    migration_class = Class.new(Gitlab::Database::Migration[1.0]) do
      include Gitlab::Database::PartitioningMigrationHelpers
    
      def up
        enqueue_partitioning_data_migration :projects
      end
    
      def assert_table_is_allowed(table_name)
        # nothing
      end
    end
    
    migration_class.new.up
  3. Create and run a temporary migration to finish the backfill:
    migration_class = Class.new(Gitlab::Database::Migration[1.0]) do
      include Gitlab::Database::PartitioningMigrationHelpers
    
      def up
        finalize_backfilling_partitioned_table :projects
      end
    
      def assert_table_is_allowed(table_name)
        # nothing
      end
    end
    
    migration_class.new.up
  4. Verify the migration has completed:
    select count(*) from projects where id not in (select id from projects_part_2c7d399fd8); -- should be 0
    select status, count(*) from background_migration_jobs where class_name = 'Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable' and arguments->>2 = 'projects' group by 1; -- they should all have status of 1

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Patrick Bair

Merge request reports

Loading