Fixing specs for single-db-two-connections

What does this MR do and why?

GitLab can be configured to connect to the PostgreSQL database in 3 different ways:

1. single-db The currently default one. Any time a customer installs GitLab, this will be the default option.

Where the config/database.yml looks something like this (without ci)

development:
  main:
    database: gitlabhq_test
    host: /Users/omar/gdk/postgresql

We are testing this setup in all the CI jobs that have single-db in the name. Where DECOMPOSED_DB environment variable is set to FALSE

2. multiple databases. That's the default for the rest of the jobs. DECOMPOSED_DB is set to true. GitLab connects to two databases main and ci. It reads/writes gitlab_main tables on the main database, and the gitlab_ci tables on the ci database. For more about this, you can read this page: https://docs.gitlab.com/ee/development/database/multiple_databases.html

We are running our SaaS gitlab.com both production and staging in this mode since summer 2022. The config/database.yml would look like this

development:
  main:
    database: gitlabhq_test
    host: /Users/omar/gdk/postgresql
  ci:
    database: gitlabhq_test_ci
    host: /Users/omar/gdk/postgresql

3. We will start supporting a new mode called single-db-ci-connection, where GitLab connects only to 1 database, but reads/writes both gitlab_main and gitlab_ci through different connections. The config/database.yml would look like this.

development:
  main:
    database: gitlabhq_test
    host: /Users/omar/gdk/postgresql
  ci:
    database: gitlabhq_test
    host: /Users/omar/gdk/postgresql
    database_tasks: false

We are switching all new and existing GitLab installations, that are not running in mode (2) multiple databases to (3) by default starting from %16.0. That's why for now we are adding jobs to test GitLab under this new mode, with the goal to get rid of all the single-db jobs later.

NOTE: The geo is outside the scope of this MR. The existence of absense of geo doesn't change the mode of the setup.

Context

In this MR and its follow up fix MR we added new jobs to test database migration and background_migration with new the new database setup single-db-ci-connection that is explained above.

Every time we run the test jobs with (1) single-db, we also run those migration and background_migration tests with (3) single-db-two-connections.

In this MR we:

1. Are fixing all the other unit, integration and system/feature tests to work with single-db-ci-connection.
2. Are adding these type of tests to run in our CI pipeline every time we run the tests against single-db. For example, if we add the label pipeline:run-single-db to a pipeline, both single-db and single-db-ci-connection jobs will run.

Rspec Helpers

Helper Name	When tests run	Status
skip_if_shared_database(:ci)	Runs tests on multiple databases	Added in this MR
skip_if_database_exists(:ci)	Runs tests on single-db, single-db-ci-connection	Added in this MR
skip_if_multiple_databases_are_setup(:ci)	Runs the tests only on single-db	Existing (not changed)
skip_if_multiple_databases_not_setup(:ci)	Runs the tests on single-db-ci-connection, multiple databases	Existing (not changed)

How to set up and validate locally

1. Make sure your TEST environment is running with single-db with ci-connection. Your config/database.yml should look contain something like this:

test:
  main: &main
    adapter: postgresql
    encoding: unicode
    database: gitlabhq_test
    host: /Users/omar/gdk/postgresql
    port: 5432
    pool: 10
    prepared_statements: false
    variables:
      statement_timeout: 120s
  ci:
    !!merge <<: *main
    database_tasks: false

2. Run rspec against these test files

ee/spec/models/factories_spec.rb
spec/lib/backup/database_spec.rb
spec/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables_spec.rb
spec/lib/gitlab/database/migration_helpers/restrict_gitlab_schema_spec.rb
spec/lib/gitlab/database/migrations/runner_spec.rb
spec/lib/gitlab/database/partitioning_spec.rb
spec/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection_spec.rb
spec/lib/gitlab/database/tables_locker_spec.rb
spec/lib/gitlab/database/tables_truncate_spec.rb
spec/lib/gitlab/database_spec.rb
spec/tasks/dev_rake_spec.rb
spec/tasks/gitlab/background_migrations_rake_spec.rb
spec/tasks/gitlab/db/decomposition/rollback/bump_ci_sequences_rake_spec.rb
spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb
spec/tasks/gitlab/db_rake_spec.rb
spec/workers/background_migration/ci_database_worker_spec.rb
spec/workers/database/batched_background_migration/ci_database_worker_spec.rb

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #391646 (closed)