Skip to content

Add Bulk Import file download & decompression services

What does this MR do?

This MR adds file download & decompression services that are going to be used during group import process when using Group Migration tool (https://docs.gitlab.com/ee/user/group/import/).

A .ndjson.gz file is going to be downloaded from a recently added 'group relations export API' (https://docs.gitlab.com/ee/api/group_relations_export.html) in order to import its contents into a group. In order to achieve this, we need to be able to safely download and decompress mentioned files.

File download service performs several validations:

  1. Do a HEAD request to /export_relations/download endpoint to validate it's content-length & content-type headers. Make sure they are acceptable.
  2. Start downloading a file by streaming its contents. If during download process file size exceeds the limit - raise exception and remove the file.

File decompression service:

  1. Decompresses a .gz file (similar to what we do in Import/Export but without tar)
  2. Performs decompressed file size validation to make sure that decompressed file size is within limits.

Both services are not yet used in this MR and are going to be used in the future MR (next one in #329864 (closed) series).

Mentions #329864 (closed)

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports

Loading