Skip to content

Update gitlab:db:configure for multiple databases

Patrick Bair requested to merge pb-configure-task-for-multiple-dbs into master

What does this MR do and why?

Related issue: #352770 (closed)

Update gitlab:db_configure to work with multiple databases. If only a single database is configured, or the ci database has database_tasks: false, it should configure the main database only.

If both databases are configured and set to database_tasks: true, it configures both based on their current status, by either loading the schema or migrating the existing schema. It only seeds the database if both databases were loaded from the schema.

We also have to consider geo, which would have database_tasks: true, but we don't want to explicitly configure (to maintain consistency with how the task previously worked). If geo is configured, we still use single-database operations like db:migrate:main to target only the databases we should setup.

How to set up and validate locally

To test with an existing schema, this can be run against your current databases. First we can test with both main and ci configured to point at the same database:

  1. Create a new migration rails g migration TestMigration with the following code:
    # frozen_string_literal: true
    
    class TestMigration < Gitlab::Database::Migration[1.0]
      def up
        say "running migration up for #{connection.pool.db_config.name}"
      end
    
      def down
        say "running migration down for #{connection.pool.db_config.name}"
      end
    end
  2. Ensure in your database.yml that the ci connection points at the same database as main (gitlabhq_development) and it has set database_tasks: false
  3. Run the configure tasks rails gitlab:db:configure, and you'll see the migration run for main only:
    == 20220322133328 TestMigration: migrating ====================================
    -- running migration up for main
    == 20220322133328 TestMigration: migrated (0.0000s) ===========================
  4. Rollback the migration rails db:rollback (or rails db:rollback:main if you have Geo configured) to test the next part
  5. Create a test database: create database configure_test;
  6. Have ci point at a the new database in database.yml and configured with database_tasks: true
  7. Run the configure tasks rails gitlab:db:configure, and you'll still see the migration run for main only:
    == 20220322133328 TestMigration: migrating ====================================
    -- running migration up for main
    == 20220322133328 TestMigration: migrated (0.0000s) ===========================
  8. In the database, check the schema was loaded into the configure_test database that ci was pointing at:
    pbair@[local]:5432/configure_test# \d
                                          List of relations
     Schema |                          Name                           |       Type        | Owner
    --------+---------------------------------------------------------+-------------------+-------
     public | abuse_reports                                           | table             | pbair
     public | abuse_reports_id_seq                                    | sequence          | pbair
     public | agent_activity_events                                   | table             | pbair
     public | agent_activity_events_id_seq                            | sequence          | pbair
     public | agent_group_authorizations                              | table             | pbair
     public | agent_group_authorizations_id_seq                       | sequence          | pbair
     public | agent_project_authorizations                            | table             | pbair
     public | agent_project_authorizations_id_seq                     | sequence          | pbair
     ...

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Patrick Bair

Merge request reports

Loading