GitLab Migration - retry BulkImports::Pipeline::Runner for additional error types
Problem
BulkImports::Pipeline::Runner
does not retry on errors that it could, making the whole migration less resilient.
Context
Failures that occur in the extract
step due to an HTTP error are being retried depending on the HTTP response error, and [GitLab Migration] Add more retriable exception... (!99275 - merged) is adding more exceptions to the list.
However, other errors could be retried since they occur in the extract
phase and didn't change the DB
Proposed solution
Retry / re-enqueue the pipeline for other types of errors, for example:
- Errors related to decompressing tar.gz files
- Lack of disk space
Technical detail
Currently, pipelines that use GraphQL perform the pagination in the same job and using recursion, which combined with the rescue StandardError causes the exception not to bubble up.
Changing the pagination to be performed in a separate job will make the code cleaner when implementing the proposed solution.