[VSA] Data collector job for the new tables
What
Implement a background job or a service that collects all stage-related data for a given group and upserts the rows in the new VSA tables (#335386 (closed))
How
High-level steps:
- Load the group (top-level groups only)
- Verify the license
- Load all configured stages for the group and its subgroups (and for projects -> not implemented yet)
- Group the stages by their hashcode (implemented in #335291 (closed))
- Select the issue related stages (inspect
GroupStage.start_event.object_type
) - Iterate over the issues in batches (using #335388 (closed))
- For each batch, load the start and end event columns defined in the issue related stages
- Transform the data to the new DB row structure (#335386 (closed))
- Upsert the data
- Do the same thing for merge request related stages.
Edited by Adam Hegyi