Skip to content

Add migration to backfill project_id for packages dependencies

Context

The packages_dependencies table currently is shared for all packages. That needs to be changed in order to have a project_id sharding key.

In Add project_id column to packages_dependencies ... (!160830 - merged) we added the project_id column to the packages_dependencies table and in Set project_id when creating Packages::Dependency (!161039 - merged) we set the project_id for new records.

What does this MR do and why?

Packages::Package is linked to the Packages::Dependency via Packages::DependencyLink. Since the Packages::Dependency currently are shared across all namespaces there're much more entries in the packages_dependency_links than in the packages_dependencies table (90M vs. 2M).

This MR adds the background migration that does the following:

  1. Sets the project_id of distinct packages_dependencies.
  2. Creates the new records in packages_dependencies for packages_dependency_links that have existing packages_dependencies with not matched project_id.
  3. Updates the packages_dependency_links with the new dependency_id from records created in (2.).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

No.

How to set up and validate locally

  1. In rails console create a couple of packages_dependencies without project_id

    # stub file upload
    def fixture_file_upload(*args, **kwargs)
      Rack::Test::UploadedFile.new(*args, **kwargs)
    end
    
    package = FactoryBot.create(:npm_package, project: Project.first)
    
    2.times do |i|
      FactoryBot.create(:packages_dependency, name: 'axios', version_pattern: "~#{i}.0.0", project: nil).tap do |dependency|
        FactoryBot.create(:packages_dependency_link, package_id: package.id, dependency_id: dependency.id, project_id: package.project_id)
      end
    end
  2. Run the background migration (make sure sidekiq is up and running)

    rails db:migrate:main

    The status of background migration may be checked in the UI http://gdk.test:3000/admin/background_migrations

  3. Verify that the project_id of packages_dependencies was set.

    Packages::Dependency.limit(2).order(id: :desc).pluck(:project_id)

Related to #465276 (closed)

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading