Skip to content

Add BulkImports Project Members Migration & Refactor Group Members Migration

George Koltsov requested to merge georgekoltsov/project-migration-members into master

GitLab Migration feature (https://docs.gitlab.com/ee/user/group/import/) is behind 2 feature flags:

  • Group Migration - bulk_import feature flag. Default enabled (& enabled on .com)
  • Project Migration - bulk_import_projects feature flag. Default disabled (& disabled on .com)

Mentions #341886 (closed)

What does this MR do and why?

This MR:

  • Adds Project Membersip migration, which preserves user memberships from source to destination project (for users with public email info)
  • Updates Group Membership migration, to not only import direct memberships, but also handle inherited memberships
  • Updates MembersPipeline & GetMemberQuery GraphQL query, which is used to fetch membership information from source, to be used in both Group & Project Migrations
  • Updates all bulk_imports/graphql/* queries to become classes, instead of modules, in order to allow easier implementation of GetMemberQuery that caters for both groups and projects (since query differs)

The entire members pipeline flow is split into 3 steps (Extract/Transform/Load) and is the following:

Data extraction

  1. For each group/project that needs to be imported, fetch direct & inherited memberships using GraphQL (GetMembersQuery)

Data transformation 🖌

For each user membership:

  1. Clean up any prohibited attributes
  2. Map public_email information on source to a user on destination, using any confirmed email
  3. If user is found, cache source_user_id => destination_user_id in redis for future use

Data load 💾

  1. Create user membership if there is no existing membership
  2. Create user membership if there is existing membership with lower access level (e.g. a user has Developer inherited membership and has Maintainer direct memberhip).
  3. Do not create user membership if existing inherited membership has higher access level
  4. Do not create user membership if user_id is current user id, since current user is already a group/project member

Screenshots or screen recordings

https://monosnap.com/file/QeXr6NeR83o76jTKCqHnyuh7BObujE

How to set up and validate locally

  1. Feature.enable(:bulk_import).
  2. Feature.enable(:bulk_import_projects).
  3. Create a top level group and project in it.
  4. Invite group and project members with public email information (as well as without)
  5. Go to /groups/new#import-group-pane page and enter instance url and access token (needs to be api & read_repository scope).
  6. Select newly created group and click Import.
  7. Wait for Group import to complete and verify imported group & project members. Users with public emails should be present with the same access levels1

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports

Loading