Skip to content

Setup new CI DB (decomposition)

Thong Kuah requested to merge ci_db into master

What does this MR do?

Related issue: #333527 (closed), &6168 (closed)

This merge request will have no effect in production, and it also will not effect local development, yet. Right now it allows you to add some extra detail to config/database.yml that allows the groupsharding to continue developing on multiple databases. Without this config specified it will do nothing.

The groupsharding is working on splitting out the CI tables to live on a separate database. Rails has support for connecting to multiple different databases at the model level. This MR does 2 things:

  1. Create a new ci database config known to Rails
  2. Create a ci_instance_variables table on this new ci database (using a migration)
  3. Allow Rails to dynamically choose whether to use the existing "primary" (or main) database or we can use the new ci configured database for this the Ci::InstanceVariable model. This should not matter which is used because the schema is the same on both

Importantly what we aren't doing yet:

  1. Actually requiring developers to configure the ci database for local development and indeed we won't be using this in production just yet
  2. Migrating the data. We don't actually plan to ever migrate this data since the table is empty on GitLab.com and using the new ci config will be optional. The migration process will actually happen later in bulk for all CI tables &6234 (closed) once we get to the point where all of the tables exist on this new database.

Links

Follow-ups

  • [-] Set up new CI DB on CI => #334233 (closed)
  • [-] Something to ensure migrations that modify ci_instance_variables table DDL, or add data to ci_instance_variables is prohibited in db/migrate => !64486 (closed)
  • [-] CI job to check CI DB structure == Primary DB structure for each table in CI DB => !64486 (closed)
  • [-] Make db/schema_migrations/ aware of multiple databases, or even better contribute this back to Rails => #334232 (closed)

Testing

DB structure load

bundle exec rails db:structure:load still works.

Also, tested with bundle exec rails db:structure:load:primary.

\dt
(506 rows)

gitlabhq_development_zzzzz=# select count(1) from schema_migrations;
 count 
-------
  2203
(1 row)

And with bundle exec rails db:structure:load:ci


# \dt
               List of relations
 Schema |         Name          | Type  | Owner 
--------+-----------------------+-------+-------
 public | ar_internal_metadata  | table | tkuah
 public | ci_instance_variables | table | tkuah
 public | schema_migrations     | table | tkuah
(3 rows)

gitlabhq_development_ci=# select * from schema_migrations
;
    version     
----------------
 20210617101848
(1 row)

DB/connection switching

Just one db configured in config/database.yml:

[1] pry(main)> Ci::InstanceVariable.connection.execute('select current_database();').to_a
   (0.4ms)  select current_database(); /*application:console,line:(pry):1:in `__pry__'*/
=> [{"current_database"=>"gitlabhq_development"}]

ci and primary db configured:

[1] pry(main)> Ci::InstanceVariable.connection.execute('select current_database();').to_a
   (0.4ms)  select current_database(); /*application:console,line:(pry):1:in `__pry__'*/
=> [{"current_database"=>"gitlabhq_development_ci"}]

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Edited by Thong Kuah

Merge request reports