Add importer to get GitHub gists and schedule worker to import each gist

What does this MR do and why?

This is the second part of the implementation for GitHub Importer: API for bulk import of personal gists to personal snippets

Right now it is blocked by !103761 (merged) which needs to be merged first.

From the original issue Problem to solve:

Problem to solve

We can import repos from GitHub, but for a more complete migration away from GitHub, we would need to have a way to import GitHub Gists as well.

With the release of Snippets with multiple Files, it is now possible to migrate GitHub Gists (up to 10 files) directly into GitLab Snippets. As we haven't had any requests to increase that limit in the product and allow snippets to have more files, we won't try to make that possible with import either.

All workers and importers are added under gitlab/github_gists_import namespace since gitlab/github_import is all about project import.

The overall feature implementation is split into 4 parts that need to be merged in sequential order:

1. Merged MR1 adds the ImportGistWorker that performs the import of one GitHub gist into a GitLab snippet and notifies JobWaiter upon importing. It calls GistImporter that is responsible to create a Snippet from gist object represented by Representation::Gist. It returns an error when the imported gist has more than 10 files.
2. This MR2 adds GistsImporter to fetch all gists from GitHub and schedules workers (from MR1) for each gist to be imported. FinishImportWorker completes the import process by setting the status as finished. It returns next_attempt_in if rate limit was reached.
3. MR3 - Add possibility to import all gists adds possibility to import all gists. Import::Github::GistsImportService is responsible for scheduling StartImportWorker based on the import status that is stored in Redis and is kept for 24 hours. This worker executes GistsImporter (from MR2) to fetch all gists from GitHub and schedules worker (from MR1) for each gist to be imported. FinishImportWorker completes the import process by setting the status as finished.
4. MR4 - Add import all gists endpoint adds new public API endpoint and documentation for it.

Diagram

graph TD
1(Start) -->|Send request| 3[POST /import/github/gists]
3 --> 4[GistsImportService]
subgraph MergeRequest3
4 -->|Check status| 6{status == 'started'}
6 -->|Yes| 7[Return 422 error]
6 -->|No| 8[StartImportWorker]
subgraph MergeRequest2
  8 -->|Log and execute importer| 9[GistsImporter]
  9 --> A[Create Waiter]
  A -->|Fetch Gists from Github| B[GithubImport::Client]
  B -->|Schedule Bulk Perform| C[ImportGistWorker]
  subgraph MergeRequest1
  C -->|Log and execute importer| D[GistImporter]
  D -->|Check execution result| E{Success?}
  E -->|Yes| F[Notify Waiter]
  E -->|No| G[Log Error]
  G --> F
  D -->|check if more than 10 files| H{valid?}
  H -->|Yes| I[Create Snippet]
  H -->|No| J[Return Error]
  end
end
9 -->|Check execution result| 10{Success?}
10 -->|Yes| 11[FinishImportWorker]
11 -->|Check Waiter| 12{jobs_remaining = 0}
12 -->|Yes| 13[Set Status to 'finished']
13 --> 14(End)
12 -->|No| 11
10 -->|No| 15{Rate limit reached?}
15 -->|Yes| 8
15 -->|No| 16[Log and raise error]
end

Screenshots or screen recordings

On GitHub I have 8 gists (1 is corrupted and 1 has more than 10 files)

so 6 gists should be imported

Screen_Recording_2022-12-07_at_12.10.43-1

How to setup and validate locally

1. From the Rails console run Gitlab::GithubGistsImport::Importer::GistsImporter.new(<user>, <personal_github_access_token>).execute
2. Observe the Snippets dashboard, imported snippets should appear.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.