Update Direct Transfer to support import in batches
What does this MR do and why?
This MR updates Direct Transfer and iterates over !124085 (merged) to start utilizing batched export on import side & import group/project relations in batches, if source instance supports batched export (16.2 onwards).
It introduces a number of new workers in order to achieve this:
-
PipelineBatchWorker
- responsible for importing a batch of group/project relation (e.g. 1000 issues of a project) -
FinishBatchedPipelineWorker
- responsible for updating parentpipeline_tracker
object after all batches are complete in order to progress with the import - Updates
PipelineWorker
to enqueue batch jobs if source export is batched (it is from 16.2 onwards)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
Migration doesn't use batch_trackers and executes entire relations within a single job |
Migration uses batch_trackers and executes relations import in batches |
batched_migration_disabled.mov | batched_migration_enabled.mov |
How to set up and validate locally
- Have a group and project structure on .com (.com already supports batched export)
- Ger personal access token from .com and initiate a new import from .com to local gdk
- Enable
:bulk_imports_batched_import_export
feature flag - Start a new direct transfer from .com to gdk
POST http://gdk.test:3000/api/v4/bulk_imports
PRIVATE-TOKEN: TOKEN
Content-Type: application/json
{
"configuration": {
"url": "https://gitlab.com/",
"access_token": "TOKEN"
},
"entities": [
{
"source_full_path": "georgekoltsov-group",
"source_type": "group_entity",
"destination_slug": "georgekoltsov-group",
"destination_namespace": "",
"migrate_projects": true
}
]
}
- Verify migration is successful
- Disable
:bulk_imports_batched_import_export
feature flag and start a new migration - Verify migration is successful and there are no entries in
bulk_import_batch_trackers
table
Mentions #391224 (closed)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by George Koltsov