Skip to content

Add new method to distribute GitHub import jobs

Rodrigo Tomonari requested to merge rodrigo/github-importer-handle-rate-limit into master

What does this MR do and why?

The current implementation of spread_parallel_import does not handle the cases where the import job is executed multiple times due to Sidekiq interruption or API rate limit errors. And because of that, records are not imported. See more information in #378906 (comment 1230743723)

This change introduces a new method for spreading the jobs that should handle Sidekiq restarts gracefully.

Related to: #378906 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Enable GitHub Import in the settings (Admin -> Settings -> General -> Visibility and access controls -> Enable GitHub)

  2. Enable the feature flag improved_spread_parallel_import 

bin/rails c

Feature.enable(:improved_spread_parallel_import)
  1. Trigger an import via API or UI
curl --location --request POST 'http://gdk.test:3000/api/v4/import/github' \
--header 'Authorization: Bearer <GITLAB ACCESS TOKEN>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "personal_access_token": "<GITHUB ACCESS TOKEN>",
    "repo_id": "238972",
    "target_namespace": "root",
    "new_name": "rspec-core",
    "optional_stages": {
        "single_endpoint_issue_events_import": true,
        "single_endpoint_notes_import": true,
        "attachments_import": false
    }
}'
  1. Monitor the import status via API or UI
curl --silent --location --request GET 'http://gdk.test:3000/api/v4/projects/<PROJECT ID>/import/' \
--header 'Authorization: Bearer <GITLAB ACCESS TOKEN>' | jq

  1. During the import process, restart Sidekiq a couple of times, or wait for the API rate limit to be reached.

  2. Check if all data was imported

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports