Skip to content

Import group epics via ndjson when using Bulk Import

George Koltsov requested to merge georgekoltsov/bulk_import_ndjson_epics into master

What does this MR do?

Depends on !62409 (merged)

This MR replaces the way we import group epics when using Bulk Import.

More information on Bulk Imports group migration tool https://docs.gitlab.com/ee/user/group/import/

Majority of Bulk Import ETL pipelines (extract -> transform -> load) use GraphQL API to import data. However, due to challenges described in #326757 (closed) not all group relations can be transferred over while preserving all their associations (e.g. if an epic has notes, which have award emojis. such nested relations are difficult to preserve using GraphQL API, due to nested pagination).

Instead, download exported epics.ndjson.gz file from 'Group relations export API' that was recently added as part of #329864 (closed) https://docs.gitlab.com/ee/api/group_relations_export.html and import it. This way we can easily preserve all nested associations, as we're reusing alot of the behaviour from existing Import/Export codebase.

Updated Epics utilises existing Import/Export RelationFactory which brings a lot of benefits, like making sure all nested relations are transformed into objects, all attributes are sanitized, appropriate attributes are added, etc

As a side effect we can remove a bunch of epics import code because of it.

Mentions #329864 (closed)

Screenshots (strongly suggested)

epicsndjson

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports

Loading