Skip to content

Add worker to wait for project export relations to finished

Rodrigo Tomonari requested to merge rodrigo/track-relation-exports into master

What does this MR do and why?

Note: The user can't trigger these workers yet. In a future MR, a feature flag will be added to enable these workers.

Related to: #366828 (closed)

This change is part of a sequence of changes that will export projects using multiple Sidekiq Jobs rather than running the whole export process in just one single job.

This MR adds the CreateRelationExportsWorker worker, which is responsible for creating a relation_export record for each project relation that needs to be exported. For context, the relation_export record is just an entry in the DB that stores information about the status of the relation export. Then for each record that is created, a RelationExportWorker worker is enqueued to initiate the export of the relation. And finally, the CreateRelationExportsWorker enqueues the WaitRelationExportsWorker worker, which will wait for all relations to be exported.

The WaitRelationExportsWorker worker is responsible for monitoring the relation exports. If one of the relation_export fails or gets stuck (the respective worker gets interrupted multiple times, and it's sent to the dead queue), the WaitRelationExportsWorker marks the whole project export as failed and notifies the error. And in case all relation_export succeeds, it calls the ParallelProjectExportWorker worker, which will merge each individual exported relation in a single archive file (project.tar.gz).

sequenceDiagram
    autonumber

    participant export_service as CreateRelationExportsWorker
    participant relations_worker as RelationExportWorker
    participant tracker as WaitRelationExportsWorker
    participant export_worker as ParallelProjectExportWorker

    par Generate relations in parallel
        export_service->>relations_worker: Enqueue worker to generate relation A
        export_service->>relations_worker: Enqueue worker to generate relation B
        export_service->>relations_worker: Enqueue worker to generate relation C
        export_service->>relations_worker: ...
    end

    export_service->>tracker: Enqueues WaitRelationExportsWorker
    loop
        tracker->>tracker: Monitor if all relations were generated. <br />Keep re-enqueuing itself until all relations are generated.<br />Notify the user in case of an error occurs
    end
 
    tracker->>export_worker: Enqueue worker

Screenshots or screen recordings

Successful export example

export-success

Export with an unexpected error

export-with-error

How to set up and validate locally

Because these workers aren't reached via UI or API, we need to use the Rails console to test them.

bin/rails c

project_id = 1
user_id = User.first.id
Projects::ImportExport::CreateRelationExportsWorker.perform_async(user_id, project_id)

Then check if the project was exported successfully by going to the Project page -> Settings -> General -> Advanced -> Export project

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports