ci: Parallelize the 'rspec foss-impact' job
What does this MR do and why?
This MR parallelize the rspec foss-impact
job, by creating a dynamic child pipeline where the parallelization count is dynamically calculated based on the number of test files to run.
In scripts/generate-rspec-foss-impact-pipeline
, we calculate the parallelization count so that each job finishes in 10 minutes (ideally), based on the current average duration of test files.
The process is as follows:
- The
rspec-foss-impact:pipeline-generate
job depends ondetect-tests
and generate the CI config of the dynamic child pipeline - The
rspec-foss-impact:trigger
job triggers the child pipeline using the CI config generated byrspec-foss-impact:pipeline-generate
.
- It depends on
setup-test-env
,retrieve-tests-metadata
,compile-test-assets as-if-foss
, andrspec-foss-impact:pipeline-generate
.
Things to note
I've split the .gitlab-ci.yml
file into dedicated files under .gitlab/ci/shared/
so that I can include these where needed in child pipeline config.
No behavior change is expected compared to the existing pipeline behavior.
Screenshots or screen recordings
Parent pipeline | Trigger job | Child pipeline |
---|---|---|
How to set up and validate locally
Test case | Job |
---|---|
No RSpec file to run | https://gitlab.com/gitlab-org/gitlab/-/jobs/3074822872 |
Less than 2 nodes to run (no parallelization) | https://gitlab.com/gitlab-org/gitlab/-/jobs/3074886292 => https://gitlab.com/gitlab-org/gitlab/-/pipelines/648926956 |
Maximum parallelization | https://gitlab.com/gitlab-org/gitlab/-/jobs/3074315058 => https://gitlab.com/gitlab-org/gitlab/-/pipelines/648828499 |
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #372343 (closed).