Fixing specs for single-db-two-connections
What does this MR do and why?
GitLab can be configured to connect to the PostgreSQL database in 3 different ways:
-
single-db
The currently default one. Any time a customer installs GitLab, this will be the default option.
Where the config/database.yml
looks something like this (without ci)
development:
main:
database: gitlabhq_test
host: /Users/omar/gdk/postgresql
We are testing this setup in all the CI jobs that have single-db
in the name. Where DECOMPOSED_DB
environment variable is set to FALSE
- multiple databases. That's the default for the rest of the jobs.
DECOMPOSED_DB
is set totrue
. GitLab connects to two databasesmain
andci
. It reads/writesgitlab_main
tables on themain
database, and thegitlab_ci
tables on theci
database. For more about this, you can read this page: https://docs.gitlab.com/ee/development/database/multiple_databases.html
We are running our SaaS gitlab.com
both production and staging in this mode since summer 2022. The config/database.yml
would look like this
development:
main:
database: gitlabhq_test
host: /Users/omar/gdk/postgresql
ci:
database: gitlabhq_test_ci
host: /Users/omar/gdk/postgresql
- We will start supporting a new mode called
single-db-ci-connection
, where GitLab connects only to 1 database, but reads/writes bothgitlab_main
andgitlab_ci
through different connections. Theconfig/database.yml
would look like this.
development:
main:
database: gitlabhq_test
host: /Users/omar/gdk/postgresql
ci:
database: gitlabhq_test
host: /Users/omar/gdk/postgresql
database_tasks: false
We are switching all new and existing GitLab installations, that are not running in mode (2) multiple databases
to (3)
by default starting from %16.0. That's why for now we are adding jobs to test GitLab under this new mode, with the goal to get rid of all the single-db
jobs later.
NOTE: The geo
is outside the scope of this MR. The existence of absense of geo
doesn't change the mode of the setup.
Context
In this MR and its follow up fix MR we added new jobs to test database migration
and background_migration
with new the new database setup single-db-ci-connection
that is explained above.
Every time we run the test jobs with (1) single-db
, we also run those migration
and background_migration
tests with (3) single-db-two-connections
.
In this MR we:
- Are fixing all the other
unit
,integration
andsystem/feature
tests to work withsingle-db-ci-connection
. - Are adding these type of tests to run in our CI pipeline every time we run the tests against
single-db
. For example, if we add the label pipeline:run-single-db to a pipeline, bothsingle-db
andsingle-db-ci-connection
jobs will run.
Rspec Helpers
Helper Name | When tests run | Status |
---|---|---|
skip_if_shared_database(:ci) | Runs tests on multiple databases | Added in this MR |
skip_if_database_exists(:ci) | Runs tests on single-db, single-db-ci-connection | Added in this MR |
skip_if_multiple_databases_are_setup(:ci) | Runs the tests only on single-db | Existing (not changed) |
skip_if_multiple_databases_not_setup(:ci) | Runs the tests on single-db-ci-connection, multiple databases | Existing (not changed) |
How to set up and validate locally
- Make sure your
TEST
environment is running withsingle-db
withci-connection
. Yourconfig/database.yml
should look contain something like this:
test:
main: &main
adapter: postgresql
encoding: unicode
database: gitlabhq_test
host: /Users/omar/gdk/postgresql
port: 5432
pool: 10
prepared_statements: false
variables:
statement_timeout: 120s
ci:
!!merge <<: *main
database_tasks: false
- Run rspec against these test files
ee/spec/models/factories_spec.rb
spec/lib/backup/database_spec.rb
spec/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables_spec.rb
spec/lib/gitlab/database/migration_helpers/restrict_gitlab_schema_spec.rb
spec/lib/gitlab/database/migrations/runner_spec.rb
spec/lib/gitlab/database/partitioning_spec.rb
spec/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection_spec.rb
spec/lib/gitlab/database/tables_locker_spec.rb
spec/lib/gitlab/database/tables_truncate_spec.rb
spec/lib/gitlab/database_spec.rb
spec/tasks/dev_rake_spec.rb
spec/tasks/gitlab/background_migrations_rake_spec.rb
spec/tasks/gitlab/db/decomposition/rollback/bump_ci_sequences_rake_spec.rb
spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb
spec/tasks/gitlab/db_rake_spec.rb
spec/workers/background_migration/ci_database_worker_spec.rb
spec/workers/database/batched_background_migration/ci_database_worker_spec.rb
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #391646 (closed)