Skip to content

Add project export relations models to Import/Export

Rodrigo Tomonari requested to merge rodrigo/360683-add-relation-exports-model into master

What does this MR do and why?

In order to parallelize the project export in Import/Export, two new models are being added to Import/Export to track and store the partial export files (relations).

The table project_relation_exports will hold the status of the relation exportation. And the table project_relation_export_uploads will store the exported file's location.

Note that this change only adds the tables and models. Other changes will be added to use these models. For example the following merge request

Related to: Add/reuse models to export top level project re... (#360683 - closed)

Context

Currently, ProjectExportService generates the project export tarball in a single process that depending on the project size, can take hours to complete. Because this solution has performance issues, the generation of the project export will be split into multiple processes. Hence these models and tables will this new approach.

The plan is to generate each part of the project export tarball in different processes and saves the partial files temporarily in the object store, later combining them in a final exported file (tarball).

Screenshots or screen recordings

bin/rails db:migrate

main: == 20220619182308 CreateProjectRelationExports: migrating =====================
main: -- create_table(:project_relation_exports, {})
main: -- quote_column_name(:relation)
main:    -> 0.0000s
main: -- quote_column_name(:jid)
main:    -> 0.0000s
main: -- quote_column_name(:export_error)
main:    -> 0.0000s
main:    -> 0.0079s
main: == 20220619182308 CreateProjectRelationExports: migrated (0.0089s) ============

main: == 20220619184931 CreateProjectRelationExportUploads: migrating ===============
main: -- create_table(:project_relation_export_uploads, {})
main: -- quote_column_name(:export_file)
main:    -> 0.0000s
main:    -> 0.0044s
main: == 20220619184931 CreateProjectRelationExportUploads: migrated (0.0045s) ======

VERSION=20220619182308 bin/rails db:migrate:down:main

main: == 20220619182308 CreateProjectRelationExports: reverting =====================
main: -- drop_table(:project_relation_exports, {})
main:    -> 0.0018s
main: == 20220619182308 CreateProjectRelationExports: reverted (0.0034s) ============

VERSION=20220619184931 bin/rails db:migrate:down:main

main: == 20220619184931 CreateProjectRelationExportUploads: reverting ===============
main: -- drop_table(:project_relation_export_uploads, {})
main:    -> 0.0255s
main: == 20220619184931 CreateProjectRelationExportUploads: reverted (0.0274s) ======

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports