Add bulk import gists endpoint
What does this MR do and why?
The third part of the implementation GitHub Importer: API for bulk import of personal gists to personal snippets
From the original issue Problem to solve:
Problem to solve
We can import repos from GitHub, but for a more complete migration away from GitHub, we would need to have a way to import GitHub Gists as well.
With the release of Snippets with multiple Files, it is now possible to migrate GitHub Gists (up to 10 files) directly into GitLab Snippets. As we haven't had any requests to increase that limit in the product and allow snippets to have more files, we won't try to make that possible with import either.
All workers and importers are added under gitlab/github_gists_import
namespace since gitlab/github_import
is all about project import.
The overall feature implementation is split into 3 parts that need to be merged in sequential order:
-
✅ Merged MR1 adds theImportGistWorker
that performs the import of one GitHub gist into a GitLab snippet and notifiesJobWaiter
upon importing. It callsGistImporter
that is responsible to create a Snippet from gist object represented byRepresentation::Gist
. It returns an error when the imported gist has more than 10 files. -
✅ Merged MR2 - Add importer to fetch gists from Github and schedule worker for each gist to be imported addsGistsImporter
to fetch all gists from GitHub and schedules workers (from MR1) for each gist to be imported.FinishImportWorker
completes the import process by setting the status as finished. -
✅ Merged MR3 - Add possibility to import all gists adds possibility to import all gists.Import::Github::GistsImportService
is responsible for schedulingStartImportWorker
based on the import status that is stored in Redis and is kept for 24 hours. This worker executesGistsImporter
(from MR2) to fetch all gists from GitHub and schedules worker (from MR1) for each gist to be imported.FinishImportWorker
completes the import process by setting the status as finished. - This MR4 adds new public API endpoint and documentation for it.
Diagram
graph TD
1(Start) -->|Send request| 3[POST /import/github/gists]
3 --> 4[GistsImportService]
subgraph MergeRequest3
4 -->|Check status| 6{status == 'started'}
6 -->|Yes| 7[Return 422 error]
6 -->|No| 8[StartImportWorker]
subgraph MergeRequest2
8 -->|Log and execute importer| 9[GistsImporter]
9 --> A[Create Waiter]
A -->|Fetch Gists from Github| B[GithubImport::Client]
B -->|Schedule Bulk Perform| C[ImportGistWorker]
subgraph MergeRequest1
C -->|Log and execute importer| D[GistImporter]
D -->|Check execution result| E{Success?}
E -->|Yes| F[Notify Waiter]
E -->|No| G[Log Error]
G --> F
D -->|check if more than 10 files| H{valid?}
H -->|Yes| I[Create Snippet]
H -->|No| J[Return Error]
end
end
9 -->|Check execution result| 10{Success?}
10 -->|Yes| 11[FinishImportWorker]
11 -->|Check Waiter| 12{jobs_remaining = 0}
12 -->|Yes| 13[Set Status to 'finished']
13 --> 14(End)
12 -->|No| 11
10 -->|No| 15{Rate limit reached?}
15 -->|Yes| 8
15 -->|No| 16[Log and raise error]
end
Screenshots or screen recordings
When import started it returns 202 Accepted
status code. If import is triggered again during import process it returns error
Screen_Recording_2022-11-08_at_09.14.40
Secret Gists become Private Snippets:
Screen_Recording_2022-11-08_at_09.16.18
Public Gists become Public Snippets
Screen_Recording_2022-11-08_at_09.15.47
If the gist have more than 10 files import is skipped and an error is logged with the message:
"message":"importer failed","error.message":"exсeeds max file limit"
How to set up and validate locally
POST to {{host}}/api/v4/import/github/gists
Request body:
{ "personal_access_token": <github_personal_access_token> }
Observe Snippets Dashboard to include imported items.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.