Add script for migrating GitLab Instance to decomposed setup

What does this MR do?

This MR introduces a new gitlab-ctl command that allows self-managed customers to upgrade their single-database GitLab instances to decomposed setup. It contains Omnibus-specific commands. The actual migration (copy the database) is part of a rake task in GitLab project.

How to test

I setup a VM on Google Cloud Compute Engine using Ubuntu Jammy and installed the gitlab package from this build

Follow steps in this this MR for 'Linux package -> Existing installations'

This will result in:

• Database gitlab_production_ci populated with data from gitlab_production
• GitLab instance using both databases

Example shell session

rwessels@glrw:~$ sudo gitlab-rails c
--------------------------------------------------------------------------------
 Ruby:         ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
 GitLab:       16.7.0-pre (1b70913da5f) EE
 GitLab Shell: 14.31.0
 PostgreSQL:   13.12
------------------------------------------------------------[ booted in 37.88s ]
Loading production environment (Rails 7.0.8)
irb(main):001:0> Ci::Pipeline.connection_db_config.database
=> "gitlabhq_production"
irb(main):002:0> Project.connection_db_config.database
=> "gitlabhq_production"
irb(main):003:0> exit

# Create new database
rwessels@glrw:~$ sudo gitlab-psql -c "CREATE DATABASE gitlabhq_production_ci WITH OWNER 'gitlab'"
CREATE DATABASE

# Backup gitlab config                                                                                                                                                                  
rwessels@glrw:~$ sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.org                                                                                                         

# Edit config but do not reconfigure
rwessels@glrw:~$ sudo vim /etc/gitlab/gitlab.rb

# Run migration script
rwessels@glrw:~$ sudo gitlab-ctl pg-decomposition-migration                                                                                                                      
This script will migrate this GitLab Instance to two-database setup                                                                                                              
                                                                                                                                                                            
Ensure:                                                                                                                                                                     
- The new database 'gitlabhq_production_ci' has been created                                                                                                                
- The following changes are added to /etc/gitlab/gitlab.rb configuration file:                                                                                              
                                                                                                                                                                            
gitlab_rails['env'] = { 'GITLAB_ALLOW_SEPARATE_CI_DATABASE' => 'true' }                                                                                                     
gitlab_rails['databases']['ci']['enable'] = true 
gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci'"

This script will:
- Disable background migrations because they should not be active during this migration
  See https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#enable-or-disable-background-migrations
- Stop the Gitlab Instance
- Copy data in gitlabhq_production to gitlabhq_production_ci (by dumping, then restoring)
- Apply configuration changes in /etc/gitlab/gitlab.rb using 'gitlab-ctl reconfigure'
- Prevent errorneous database access
- Re-enable background migrations

This script will not:
- Clean up data in the databases
- Start GitLab

Please confirm the upgrade by pressing 'y':
y
Disabling Background Migrations...
Stopping GitLab...
Copying data to new database...
Reconfigure GitLab...
Enable write locks...
Enabling Background Migrations...
GitLab is now running on two databases. Data related to CI is now written to the ci
database.

You can now start the GitLab services by running 'sudo gitlab-ctl restart'

You can also remove duplicated data by running:
'sudo gitlab-rake gitlab:db:truncate_legacy_tables:main'
'sudo gitlab-rake gitlab:db:truncate_legacy_tables:ci'

# Done! start GitLab
rwessels@glrw:~$ sudo gitlab-ctl start
ok: run: alertmanager: (pid 16404) 1s
ok: run: crond: (pid 16413) 0s
ok: run: gitaly: (pid 16421) 1s
ok: run: gitlab-exporter: (pid 16439) 0s
ok: run: gitlab-kas: (pid 16493) 0s
ok: run: gitlab-workhorse: (pid 16502) 1s
ok: run: logrotate: (pid 16510) 0s
ok: run: nginx: (pid 16516) 1s
ok: run: node-exporter: (pid 16530) 0s
ok: run: postgres-exporter: (pid 16535) 0s
ok: run: postgresql: (pid 16559) 0s
ok: run: prometheus: (pid 16562) 0s
ok: run: puma: (pid 16577) 1s
ok: run: redis: (pid 16591) 0s
ok: run: redis-exporter: (pid 16598) 0s
ok: run: registry: (pid 16605) 1s
ok: run: sidekiq: (pid 16622) 0s

# CI-related models are now using CI database
rwessels@glrw:~$ sudo gitlab-rails c
--------------------------------------------------------------------------------
 Ruby:         ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
 GitLab:       16.7.0-pre (1b70913da5f) EE
 GitLab Shell: 14.31.0
 PostgreSQL:   13.12
------------------------------------------------------------[ booted in 52.29s ]
Loading production environment (Rails 7.0.8)
irb(main):001:0> Ci::Pipeline.connection_db_config.database
=> "gitlabhq_production_ci"
irb(main):002:0> Project.connection_db_config.database
=> "gitlabhq_production"
irb(main):003:0> 

Related issues

gitlab#368729 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated. Draft MR
• Tests added.
• Integration tests added to GitLab QA. Not applicable
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file. Not applicable