Feedback issue: Next generation container registry rollout to self-managed
Next Generation Container Registry
&5521 describes in detail the motivation and technical information about the next-generation container registry. This issue is intended to provide feedback to the GitLab team from self-managed customers who have started or completed the migration to the new registry.
Overview
In Milestone 8.8, GitLab launched the MVC of the Container Registry. This feature integrated the Docker Distribution registry into GitLab so that any GitLab user could have a space to publish and share container images.
However, there was an inherent limitation with Docker Distribution, as all metadata associated with a given image/tag was stored in the object storage backend. This made using that metadata to build API features (like storage usage visibility, sorting, and filtering) unfeasible. The most recent Container Registry update added a new PostgreSQL backend, which is used to store the metadata. Additionally, this new version also includes an automatic online garbage collector to remove untagged images and recover storage space.
Helpful links
- GitLab Container Registry administration
- Metadata database administration for Linux installations
- Multistep Import Video Walkthrough
- Database Quick Start Guide
Pre Migration Checklist
- Minimum GitLab version: 17.1 — 17.3 (GA) recommended
- Make a backup of the registry's object storage before migrating to the database
- Ensure that you never run offline garbage collection after migration (check for cron jobs!)
- Be aware that all untagged images will be removed automatically by online garbage collection
- Understand once the database is enabled, it must not be disabled again unless you have restored the pre-migration object storage from backup
- Your GitLab instance does not use Geo — Geo is not yet supported, see Geo: Document Using the New Container Registry ... (#458440)
Import Time Estimates
These estimates are provided as a rough guide on import time for various size registries. Different environments may produce results that are faster or slower than these time indicates. If you use these figures in your own planning, please consider them as a lower bound and allow for longer import times.
Completed Tests and User Reports
These entries are from imports of test registries performed by GitLab team members, or from reports from users.
Type | Size | Storage Backend | Step One | Step Two | Step Three | Total |
---|---|---|---|---|---|---|
Test Registry | 100 GiB | GCS | 23 min | 2 min | 27 min | 52 min |
Test Registry | 100 GiB | S3 | 6 min | 11 sec | 1.5 min | 8 min |
User Registry | 160 GiB | S3 | 14 min | 1 min | 2 Min | 17 min |
Test Registry | 500 GiB | GCS | 2 hours | 9 min | 2.5 hours | 4.6 hours |
Test Registry | 500 GiB | S3 | 20 min | 90 sec | 6 min | 28 min |
User Registry | 14 TiB | S3 | 14.32 hours | 19 min | 51.5 Hours | 66 hours |
User Registry (before tag cleanup) | 535 TiB | NFS | 278.4 hours | not reported | not reported | not reported |
User Registry (after tag cleanup) | 535 TiB | NFS | 145 hours | 1 hour | not reported | not reported |
User Registry | 609 TiB | S3 | 344 hours | 12.5 hours | 615 hours | 970 hours |
Note: The 535 TiB GCS User Registry's step one and step two completed significantly faster after unused tags were deleted. Removing unused tags, if possible, is recommended before performing imports of large registry deployments.
Instructions - please read before posting!
Thanks for migrating to the next-generation container registry! Before leaving a comment, be sure to checkout our list of things we are already aware of and plan on addressing.
- This feedback issue is for migrating to the next-generation container registry. If you have other container registry feedback please consider opening a new issue.
- We will read all of your feedback!
- We will not respond to all feedback
- All found issues will be logged in the action summary of this issue
Actions we are taking from the feedback
- TBD
Bugs
- TBD
What we've started looking in to
This is known
-
gitlab-ctl reconfigure
removes the registry database configuration on GitLab versions older than 16.4 - rare image formats will stop the import process on GitLab version older than 16.5
Supported feature status
Feature | Description | Status | Link |
---|---|---|---|
Import Tool | To migrate existing deployments to the database. | Complete | Import Tool |
Automatic Import Validation | Tests that the import maintained data integrity of imported images. | Backlog | Validate self-managed imports |
Geo Support | Confirm that Geo replication works with the metadata database. | Backlog | Geo Support |
Migration Runtime Estimates | Estimate how long a migration should take in various situations. | In progress | Test offline migration runtime |
HA Database Support | HA support for the metadata database. | Backlog | HA support for other databases |
Import Step Retry | Retry import steps on failures to increase reliability. | Backlog | Retry Step on Transient Error |
Tag Concurrency | Allow simultaneous imports of tags to minimize downtime. | Backlog | Expose Tag Concurrency Option |
DB Provisioning(Omnibus) | Automatically provision a new database for Omnibus GitLab installs. | Scheduled | Provision Database (Omnibus) |
Lock Files | Use lock files to protect database-managed data. | Backlog | Lock Files to Preserve Data Consistency |