Add script for migrating GitLab Instance to decomposed setup
What does this MR do?
This MR introduces a new gitlab-ctl
command that allows self-managed customers to upgrade their single-database GitLab instances to decomposed setup. It contains Omnibus-specific commands. The actual migration (copy the database) is part of a rake task in GitLab project.
How to test
I setup a VM on Google Cloud Compute Engine using Ubuntu Jammy and installed the gitlab package from this build
Follow steps in this this MR for 'Linux package -> Existing installations'
This will result in:
- Database
gitlab_production_ci
populated with data fromgitlab_production
- GitLab instance using both databases
Example shell session
rwessels@glrw:~$ sudo gitlab-rails c
--------------------------------------------------------------------------------
Ruby: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
GitLab: 16.7.0-pre (1b70913da5f) EE
GitLab Shell: 14.31.0
PostgreSQL: 13.12
------------------------------------------------------------[ booted in 37.88s ]
Loading production environment (Rails 7.0.8)
irb(main):001:0> Ci::Pipeline.connection_db_config.database
=> "gitlabhq_production"
irb(main):002:0> Project.connection_db_config.database
=> "gitlabhq_production"
irb(main):003:0> exit
# Create new database
rwessels@glrw:~$ sudo gitlab-psql -c "CREATE DATABASE gitlabhq_production_ci WITH OWNER 'gitlab'"
CREATE DATABASE
# Backup gitlab config
rwessels@glrw:~$ sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.org
# Edit config but do not reconfigure
rwessels@glrw:~$ sudo vim /etc/gitlab/gitlab.rb
# Run migration script
rwessels@glrw:~$ sudo gitlab-ctl pg-decomposition-migration
This script will migrate this GitLab Instance to two-database setup
Ensure:
- The new database 'gitlabhq_production_ci' has been created
- The following changes are added to /etc/gitlab/gitlab.rb configuration file:
gitlab_rails['env'] = { 'GITLAB_ALLOW_SEPARATE_CI_DATABASE' => 'true' }
gitlab_rails['databases']['ci']['enable'] = true
gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci'"
This script will:
- Disable background migrations because they should not be active during this migration
See https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#enable-or-disable-background-migrations
- Stop the Gitlab Instance
- Copy data in gitlabhq_production to gitlabhq_production_ci (by dumping, then restoring)
- Apply configuration changes in /etc/gitlab/gitlab.rb using 'gitlab-ctl reconfigure'
- Prevent errorneous database access
- Re-enable background migrations
This script will not:
- Clean up data in the databases
- Start GitLab
Please confirm the upgrade by pressing 'y':
y
Disabling Background Migrations...
Stopping GitLab...
Copying data to new database...
Reconfigure GitLab...
Enable write locks...
Enabling Background Migrations...
GitLab is now running on two databases. Data related to CI is now written to the ci
database.
You can now start the GitLab services by running 'sudo gitlab-ctl restart'
You can also remove duplicated data by running:
'sudo gitlab-rake gitlab:db:truncate_legacy_tables:main'
'sudo gitlab-rake gitlab:db:truncate_legacy_tables:ci'
# Done! start GitLab
rwessels@glrw:~$ sudo gitlab-ctl start
ok: run: alertmanager: (pid 16404) 1s
ok: run: crond: (pid 16413) 0s
ok: run: gitaly: (pid 16421) 1s
ok: run: gitlab-exporter: (pid 16439) 0s
ok: run: gitlab-kas: (pid 16493) 0s
ok: run: gitlab-workhorse: (pid 16502) 1s
ok: run: logrotate: (pid 16510) 0s
ok: run: nginx: (pid 16516) 1s
ok: run: node-exporter: (pid 16530) 0s
ok: run: postgres-exporter: (pid 16535) 0s
ok: run: postgresql: (pid 16559) 0s
ok: run: prometheus: (pid 16562) 0s
ok: run: puma: (pid 16577) 1s
ok: run: redis: (pid 16591) 0s
ok: run: redis-exporter: (pid 16598) 0s
ok: run: registry: (pid 16605) 1s
ok: run: sidekiq: (pid 16622) 0s
# CI-related models are now using CI database
rwessels@glrw:~$ sudo gitlab-rails c
--------------------------------------------------------------------------------
Ruby: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
GitLab: 16.7.0-pre (1b70913da5f) EE
GitLab Shell: 14.31.0
PostgreSQL: 13.12
------------------------------------------------------------[ booted in 52.29s ]
Loading production environment (Rails 7.0.8)
irb(main):001:0> Ci::Pipeline.connection_db_config.database
=> "gitlabhq_production_ci"
irb(main):002:0> Project.connection_db_config.database
=> "gitlabhq_production"
irb(main):003:0>
Related issues
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
-
MR title and description are up to date, accurate, and descriptive. -
MR targeting the appropriate branch. -
Latest Merge Result pipeline is green. -
When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.
For GitLab team members
If you don't have access to this, the reviewer should trigger these jobs for you during the review process.
-
The manual Trigger:ee-package
jobs have a green pipeline running against latest commit. -
If config/software
orconfig/patches
directories are changed, make sure thebuild-package-on-all-os
job within theTrigger:ee-package
downstream pipeline succeeded. -
If you are changing anything SSL related, then the Trigger:package:fips
manual job within theTrigger:ee-package
downstream pipeline must succeed. -
If CI configuration is changed, the branch must be pushed to dev.gitlab.org
to confirm regular branch builds aren't broken.
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes. -
Documentation created/updated. Draft MR -
Tests added. -
Integration tests added to GitLab QA. Not applicable -
Equivalent MR/issue for the GitLab Chart opened. -
Validate potential values for new configuration settings. Formats such as integer 10
, duration10s
, URIscheme://user:passwd@host:port
may require quotation or other special handling when rendered in a template and written to a configuration file. Not applicable
Edited by Rutger Wessels