Add migration_plan to container_repositories
🔎 What does this MR do and why?
We are moving all container repositories to the new registry on GitLab.com. In Phase 2 of this move, we are importing existing container repositories in several stages by customer tier (or plan).
Container repositories are far removed from their respective plan: container_repositories -> projects -> namespaces -> gitlab_subscriptions -> plans
.
We need the ability to quickly query the number of repositories in each plan and see their current migration state. Joining against this many tables is not performant, so we are going to add a migration_plan
column to container_repositories
to hold the plan.name
value for easy access.
We know this value may become inaccurate over time as different groups/namespaces change tiers, but that is ok, we may refresh these values along the way, and if a small number of container repositories get imported with a certain plan that they no longer belong to, that is an acceptable outcome.
🐘 Database
Background migration
See the database testing output for the EachBatch queries run in the post-deployment migration. Since we are doing a full table iteration with no conditions on container_repositories
, there are no surprises and timings look good.
Updating a container_repository
record. Here are a few sample queries, averaging around ~50ms each:
- https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/9291/commands/32817
- https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/9291/commands/32818
- https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/9291/commands/32819
- https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/9291/commands/32820
Each job:
Batch size: 1500
Update timing: 50ms
(1500 * 50ms) / 1000 = 75 sec per job
This gives some room to spare in the 120 sec max job time target
Total migration:
Rows to be updated: 2112613
Batch size: 1500
Total batches/jobs: 2112613 / 1500 = 1409
Job Delay: 2 minutes
Total Run Time: (1409 * 2 min) / 60 min per hour / 24 hours per day = 1.96 Days
Migrations
Up:
== 20220316202200 AddMigrationPlanToContainerRepositories: migrating ==========
-- add_column(:container_repositories, :migration_plan, :text)
-> 0.0030s
== 20220316202200 AddMigrationPlanToContainerRepositories: migrated (0.0030s) =
== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: migrating =
-- transaction_open?()
-> 0.0000s
-- current_schema()
-> 0.0003s
-- transaction_open?()
-> 0.0000s
-- execute("ALTER TABLE container_repositories\nADD CONSTRAINT check_05e9012f36\nCHECK ( char_length(migration_plan) <= 255 )\nNOT VALID;\n")
-> 0.0014s
-- current_schema()
-> 0.0002s
-- execute("SET statement_timeout TO 0")
-> 0.0005s
-- execute("ALTER TABLE container_repositories VALIDATE CONSTRAINT check_05e9012f36;")
-> 0.0010s
-- execute("RESET statement_timeout")
-> 0.0005s
== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: migrated (0.0216s)
== 20220316202640 PopulateContainerRepositoriesMigrationPlan: migrating =======
-- Scheduled 0 PopulateContainerRepositoryMigrationPlan jobs with a maximum of 1500 records per batch and an interval of 120 seconds.
The migration is expected to take at least 0 seconds. Expect all jobs to have completed after 2022-03-17 01:44:01 UTC."
== 20220316202640 PopulateContainerRepositoriesMigrationPlan: migrated (0.0216s)
Down:
== 20220316202640 PopulateContainerRepositoriesMigrationPlan: reverting =======
== 20220316202640 PopulateContainerRepositoriesMigrationPlan: reverted (0.0000s)
== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: reverting =
-- transaction_open?()
-> 0.0000s
-- transaction_open?()
-> 0.0000s
-- execute("ALTER TABLE container_repositories\nDROP CONSTRAINT IF EXISTS check_05e9012f36\n")
-> 0.0013s
== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: reverted (0.0167s)
== 20220316202200 AddMigrationPlanToContainerRepositories: reverting ==========
-- remove_column(:container_repositories, :migration_plan, :text)
-> 0.0026s
== 20220316202200 AddMigrationPlanToContainerRepositories: reverted (0.0098s) =
Screenshots or screen recordings
N/A
How to set up and validate locally
- Create some container repositories locally if you do not already have some
10.times { FactoryBot.create(:container_repository, project: Project.first) }
- Change the project's tier
Plan.create!(name: 'ultimate') GitlabSubscription.create(hosted_plan_id: Plan.last.id, namespace_id: Project.first.namespace.id)
- Run the migrations locally
- Ensure the background migration sidekiq jobs have run locally (only one job should have been created unless you have over 1500 local container repositories)
- The
container_repositories
should now show the new plan:ContainerRepository.first.migration_plan => 'ultimate'
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #356218 (closed)