Move Group Migration entities import to individual sidekiq jobs
What does this MR do?
This MR:
- Changes Bulk Import (Group Migration) entities import process to be moved into individual sidekiq jobs (
BulkImports::ProcessEntityWorker
), instead of all being within one job - Adds batched & throttled entities import (arbitrary number 5 set as a default batch size for now) to prevent potential resource utilisation issues
- Changes
BulkImportWorker
to make sure that no more thanBulkImportWorker::DEFAULT_BATCH_SIZE
jobs are ran at any given point - Changes
BulkImportWorker
to add processing safeguards
To test:
- Seed your env with groups by running rake task
bundle exec rake "gitlab:seed:group_seed[3,root]"
(this might take few minutes) - Start sidekiq (having it running as part of gdk is fine).
- Start rails console and run (replace with your values)
user = User.first
credentials = { url: 'http://gdk.test:3000', access_token: 'token' }
params = [{ source_type: 'group_entity', source_name: '<source group name>', source_full_path: '<source full path>', destination_name: "destination", destination_namespace: 'root' }]
BulkImportService.new(user, params, credentials).execute
-
'token'
needs to be a Personal Access Token ofUser.first
withapi
scope (notread_api
) -
'<source group name>'
should be name of a root group that you got from seeding -
'<source group path>'
should be path of a root group that you got from seeding
- Observe sidekiq logs, importer log file and db tables to see import gradually importing all of the groups from source to destination
- Check destination group, verify all subgroups are migrated over
- To check sidekiq logs,
gdk tail rails-background-jobs
- For log file, it's
tail -f log/importer.log
- In rails console, check if
BulkImport.last.finished?
. Once it is, check your new group called 'destination' has all descendant groups and epics present.
Mentions #270098 (closed) (but not fully closes that issue).
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation (if required)
- [-] Code review guidelines
-
Merge request performance guidelines -
Style guides - [-] Database guides
- [-] Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov