Skip to content

Move Group Migration entities import to individual sidekiq jobs

George Koltsov requested to merge georgekoltsov/bulk-import-entity-worker into master

What does this MR do?

This MR:

  1. Changes Bulk Import (Group Migration) entities import process to be moved into individual sidekiq jobs (BulkImports::ProcessEntityWorker), instead of all being within one job
  2. Adds batched & throttled entities import (arbitrary number 5 set as a default batch size for now) to prevent potential resource utilisation issues
  3. Changes BulkImportWorker to make sure that no more than BulkImportWorker::DEFAULT_BATCH_SIZE jobs are ran at any given point
  4. Changes BulkImportWorker to add processing safeguards

To test:

  1. Seed your env with groups by running rake task bundle exec rake "gitlab:seed:group_seed[3,root]" (this might take few minutes)
  2. Start sidekiq (having it running as part of gdk is fine).
  3. Start rails console and run (replace with your values)
user = User.first
credentials = { url: 'http://gdk.test:3000', access_token: 'token' }
params = [{ source_type: 'group_entity', source_name: '<source group name>', source_full_path: '<source full path>', destination_name: "destination", destination_namespace: 'root' }]

BulkImportService.new(user, params, credentials).execute
  • 'token' needs to be a Personal Access Token of User.first with api scope (not read_api)
  • '<source group name>' should be name of a root group that you got from seeding
  • '<source group path>' should be path of a root group that you got from seeding
  1. Observe sidekiq logs, importer log file and db tables to see import gradually importing all of the groups from source to destination
  2. Check destination group, verify all subgroups are migrated over
  • To check sidekiq logs, gdk tail rails-background-jobs
  • For log file, it's tail -f log/importer.log
  • In rails console, check if BulkImport.last.finished?. Once it is, check your new group called 'destination' has all descendant groups and epics present.

Mentions #270098 (closed) (but not fully closes that issue).

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports

Loading