Add migration to backfill project_id for packages dependencies
Context
The packages_dependencies
table currently is shared for all packages. That needs to be changed in order to have a project_id
sharding key.
In Add project_id column to packages_dependencies ... (!160830 - merged) we added the project_id
column to the packages_dependencies table and in Set project_id when creating Packages::Dependency (!161039 - merged) we set the project_id
for new records.
What does this MR do and why?
Packages::Package
is linked to the Packages::Dependency
via Packages::DependencyLink
. Since the Packages::Dependency
currently are shared across all namespaces there're much more entries in the packages_dependency_links than in the packages_dependencies table (90M vs. 2M).
This MR adds the background migration that does the following:
- Sets the
project_id
of distinctpackages_dependencies
. - Creates the new records in
packages_dependencies
forpackages_dependency_links
that have existingpackages_dependencies
with not matchedproject_id
. - Updates the
packages_dependency_links
with the newdependency_id
from records created in (2.).
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
No.
How to set up and validate locally
-
In
rails console
create a couple ofpackages_dependencies
withoutproject_id
# stub file upload def fixture_file_upload(*args, **kwargs) Rack::Test::UploadedFile.new(*args, **kwargs) end package = FactoryBot.create(:npm_package, project: Project.first) 2.times do |i| FactoryBot.create(:packages_dependency, name: 'axios', version_pattern: "~#{i}.0.0", project: nil).tap do |dependency| FactoryBot.create(:packages_dependency_link, package_id: package.id, dependency_id: dependency.id, project_id: package.project_id) end end
-
Run the background migration (make sure
sidekiq
is up and running)rails db:migrate:main
The status of background migration may be checked in the UI http://gdk.test:3000/admin/background_migrations
-
Verify that the
project_id
ofpackages_dependencies
was set.Packages::Dependency.limit(2).order(id: :desc).pluck(:project_id)
Related to #465276 (closed)