Skip to content

Run RSpec jobs in a dynamic child pipeline [RUN ALL RSPEC] [RUN AS-IF-FOSS]

Albert Salim requested to merge 263407-dynamic-parallel-rspec into master

What does this MR do?

Run RSpec jobs in a dynamic child pipeline.

All rspec unit/migration/integration/system * jobs would be run in a child pipeline, linked to the main pipeline through bridge job rspec-child-pipeline.

The CI configuration YAML for this child pipeline is generated in create-rspec-pipeline-yaml job, which calculates the number of parallel jobs needed for the various rspec jobs, in order to keep them within a desired duration.

This would remove the need to periodically update the parallel count as the size of the test suite grows.

Caveats:

  • rspec migration jobs are manually parallelized, because of known issue in the knapsack report used.
  • rspec geo jobs are manually parallelized as there is only one job that is parallelized at the moment. Making this dynamic would require some additional changes to Quality::TestLevel test level detection. The added complexity may not be worthwhile at this point for 1 job.

Latest pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/218360090

Effect on pipeline duration

The changes here have a small reduction effect on the pipeline duration for MRs (~ 4 minutes based on measurements below), but it may be insignificant due to the variation in migration job duration.

To compare the effect of this change on the pipeline duration, we can compare this MR's pipeline with a benchmark RUN ALL RSPEC MR pipeline. We would be interested in the time it takes from the start of the pipeline to the start of rspec:coverage job, which signifies the completion of rspec jobs.

Milestones Timestamp on this MR pipeline (duration) Timestamp on benchmark MR pipeline (duration)
start of pipeline 4:58pm GMT+8 (00:00) 9:09pm GMT+8 (00:00)
start of rspec unit job 5:08pm GMT+8 (00:10) 9:18pm GMT+8 (00:09)
start of rspec:coverage job 5:51pm GMT+8 (00:43) 9:56pm GMT+8 (00:47)

MR Dependencies:

Things to verify:

  • rspec jobs in child pipelines are successful
  • rspec job durations
  • rspec:coverage job duration
  • update-test-metadata job duration
  • cross pipeline artifacts (rspec test dependencies, rspec coverage, rspec metadata)
  • RUN ALL RSPEC and RUN AS-IF-FOSS pipeline
  • Pipeline for merged results
  • Test in fork pipeline
Open questions: - [x] Q: How can we create pipeline with dynamic parallel count? - A: we can dynamically generate from template, passing in values from a ruby script - [x] Q: Creating child pipeline is successful. Some rspec jobs are failing while some are passing. Seems to be some set up related, but why is it inconsistent across jobs? !47377 (comment 445985167) - A: YAML variables take precedent over project pipeline in a child pipeine. Still need to figure out how to ensure project pipeline takes priority. - [x] Q: How can we get artifacts from jobs in child pipeline into a job in a parent pipeline? - A: API would be preferred. Using `needs` across pipeline is not guaranteed, as it always picks up latest job, not necessarily in the same pipeline context. Also, using `needs` conflicts in jobs that use DAG, as the downstream job will immediately start if a `needed` job in a child pipeline does not exist yet. Example job

Other findings:

  • Viewing pipeline takes a little more effort as we need to click on bridge job to expand child pipeline view.
  • Pipeline id in rspec jobs is child pipeline id, not parent pipeline id, so it takes additional step to navigate to view parent pipeline.

Related to #263407 (closed)

Edited by Albert Salim

Merge request reports