Skip to content

Recover model versions with destroyed packages

Alper Akgun requested to merge 461322-bug-emptying-all-the-files-in-package-registry-leaves-a-model-version-without-a-package into master

What does this MR do and why?

Recover destroyed ML model version packages

  • We had a fixed issue, where ML model version's package was deleted due to package registry cleanup policies
  • We have removed the policy and now we fix broken ml model versions

We have at 1K total records and 12 broken ones on gitlabdotcom. On self managed we can have more or less.

Related to #461322 (closed)

How to set up and validate locally

  1. Go to any project; and then to "Deploy"/"Model Registry"
  2. Create a model and a version, optionally uploading files
  3. On rails console (bin/rails c) emulate the broken state by running

For example on my URL http://gdk.test:3000/g1/p3/-/ml/models/120/versions/316#/

Ml::ModelVersion.find(316).package.destroy

Refresh the screen to see it now gives an error

image

  1. Run the batched background migration on your rails console bin/rails c by pasting the below.
Gitlab::BackgroundMigration::RecoverDeletedMlModelVersionPackages.new(
      start_id: 1,
      end_id: 1000,
      batch_table: :ml_model_versions,
      batch_column: :id,
      sub_batch_size: 100,
      pause_ms: 100,
      connection: Ml::ModelVersion.connection
    ).perform

Now refresh your model version page and all should look fine

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Edited by Alper Akgun

Merge request reports