Skip to content

Project import encounter ImportFailure when ci_pipelines contains external_pull_request

Summary

When project.json has ci_pipelines->external_pull_request and external_pull_requests, import will throw error

#<ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_external_pull_requests_on_project_and_branches"
DETAIL:  Key (project_id, source_branch, target_branch)=(1, feature, master) already exists.

This is because of: when process_relation! process ci_pipelines, it saves ci_pipelines->external_pull_request in DB already. Later when process_relation! process external_pull_requests, it tries to save the same external_pull_request into DB.

This issue is initially found when investigating rspec issue: #207208 (comment 293775018). And we had reproduced this behavior by using a GitLab export file.

It seems project usually do not have pipeline for external_pull_request, so this case is a rare case. With ImportFailure introduced, such case import does finish. But it still is an issue. It is just hidden. We could find the error message from the table ImportFailure after import finished.

We could expose it by: re-order the ci_pipelines to sit after external_pull_requests in import_export.yml file, this way, the external_pull_requests are still saved in DB, but the ci_pipelines will not be imported at all(due to the same exception).

Steps to reproduce

Using this Gitlab export file: 2020-02-25_12-10-187_alipniagov_dotfiles_export.tar.gz, run project import using the rake task: bundle exec rake "gitlab:import_export:import[root, root, rake_imported_project_test_external_pr, /Users/qingyu/Downloads/2020-02-25_12-10-187_alipniagov_dotfiles_export.tar.gz]" This will report one error:

Total number of not imported relations: 1
Done!

Now check the table import_failures from rails console, we could find the details of failure:

[35] pry(main)> ImportFailure.last
  ImportFailure Load (0.4ms)  SELECT "import_failures".* FROM "import_failures" ORDER BY "import_failures"."id" DESC LIMIT $1  [["LIMIT", 1]]
=> #<ImportFailure:0x00007fd4167e80f0
 id: 19,
 relation_index: 3,
 project_id: 17,
 created_at: Wed, 26 Feb 2020 04:53:16 UTC +00:00,
 relation_key: [FILTERED],
 exception_class: "ActiveRecord::RecordNotUnique",
 correlation_id_value: "c553c48eb072e3f05c65aa3918382a90",
 exception_message: "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\nDETAIL:  Key (project_id, source_branch, target_branch)=(17, new-branch, master) already exists.\n",
 retry_count: 0,
 group_id: nil,
 action: nil,
 source: "process_relation_item!">

Example Project

The project is: https://gitlab.com/alipniagov/dotfiles. The project export file is available 2020-02-25_12-10-187_alipniagov_dotfiles_export.tar.gz

What is the current bug behavior?

There is ImportFailure when importing such a project.

What is the expected correct behavior?

There is NO ImportFailure when importing such a project.

Possible fixes

I suspect we should handle external_pull_requests in the same way as labels and milestones, where it should find_or_create_object!. But I am not certain whether this is the right approach.

Edited by Qingyu Zhao