Skip to content

Add registry-database migrate command

Jaime Martinez requested to merge registry-database-migrations into master

What does this MR do?

This MR adds a new command to gitlab-ctl to manage schema migrations for the registry metadata database. It adds the following commands as printed by the usage output.

Usage:
  gitlab-ctl registry-database command subcommand [options]

GLOBAL OPTIONS:
  -h, --help      Usage help

COMMANDS:
  migrate                Manage schema migrations
Manage migrations

Usage:
  gitlab-ctl registry-database migrate SUBCOMMAND [options]

Subcommands:
  down        Apply down migrations
  status      Show migration status
  up          Apply up migrations
  version     Show current migration version

Options:
  -h, --help   help for migrate

The subcommand verifies that:

  1. The registry is enabled before proceeding
  2. The registry binary exists
  3. That the configuration file exists
  4. Asks the user to stop the registry for the up and down subcommands (except when used with -d, --dry-run).
  5. Applies up/down migrations as request, or
  6. Checks the migration status or version

You can think of these new command and subcommand as an alias of the registry own's database migrations available to the registry binary under https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/database-migrations.md?ref_type=heads#administration.

Why?

As of today, users in the Beta program need to follow some manual steps to apply the schema migrations. While this change also requires admins to run these commands, the former approach requires deeper knowledge of the toolset, like having to find the registry binary.

This new subcommand will abstract that from the user and direct them to a tool they are familiar with such as gitlab-ctl.

Testing

  1. Setup a development omnibus environment
  2. Provision a separate Postgres instance and create a logical database, for example:
CREATE USER registry WITH PASSWORD 'registrypassword'
CREATE DATABASE registry_omnibus WITH OWNER registry

  1. Alternatively, you can use the provisioned Postgres instance but you will need to allow the registry to connect to it via Unix socket, for example follow https://gitlab.com/-/snippets/2572204#prepare-the-metadata-database-for-the-container-registry.

  2. Configure the database section under the registry configuration, with the enabled flag set to false

registry['database'] = {
   'enabled' => false,
   'host' => '172.17.0.1', # or path to unix socket
   'port' => 5432,
   'user' => 'registry',
   'password' => 'registrypassword',
   'dbname' => 'registry_omnibus',
   'sslmode' => 'disable' # one of disable, allow, prefer, require, verify-ca or verify-full. See the PostgreSQL documentation for additional information https://www.postgresql.org/docs/current/libpq-ssl.html.
}
  1. Reconfigure gitlab-ctl reconfigure

Test the subcommands

To test the newly added subcommands, you need to replace/symlink the changed files in the right directory under your Omnibus installation. For example

ln -s /home/dev/files/gitlab-ctl-commands/registry_database.rb /opt/gitlab/embedded/service/omnibus-ctl/
ln -s /home/dev/files/gitlab-ctl-commands/lib/ /opt/gitlab/embedded/service/omnibus-ctl/

Some of the subcommands to test:

gitlab-ctl registry-database migrate -h
gitlab-ctl registry-database migrate up -h
gitlab-ctl registry-database migrate up -d # dry run will not commit changes to the DB
gitlab-ctl registry-database migrate down -h 
gitlab-ctl registry-database migrate status
gitlab-ctl registry-database migrate version
  1. When ready, run the migrate up subcommand and check for the OK: applied X migrations message
gitlab-ctl registry-database migrate up
... # long output
OK: applied 145 migrations in 19.781s
  1. Update your /etc/gitlab/gitlab.rb file and set the enabled flag to true under the registry['database'] configuration section
registry['database'] = {
   'enabled' => true,
...

  1. gitlab-ctl reconfigure

  2. That should start the registry with the metadata database enabled. Tail the registry logs and look for the message:

2023-10-20_06:03:10.22422 time="2023-10-20T06:03:10.224Z" level=warning msg="the metadata database is a beta feature, please carefully review the documentation before enabling it in production" environment=production go_version=go1.20.7 instance_id=23dad3ac-897f-4f89-9bc5-97e9867fd9e0 service=registry version=v3.83.0-gitlab

Optionally, try to push an image to the registry. That needs docker login with a private token and an existing project under gitlab. You can also connect to the provisioned database and check some tables after successfully pushing the image

select * from manifests;
select * from tags;

Related issues

Related to Add registry migration subcommand to Omnibus' g... (container-registry#1108 - closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by Jaime Martinez

Merge request reports