Skip to content

ci: Parallelize the 'rspec foss-impact' job

What does this MR do and why?

This MR parallelize the rspec foss-impact job, by creating a dynamic child pipeline where the parallelization count is dynamically calculated based on the number of test files to run.

In scripts/generate-rspec-foss-impact-pipeline, we calculate the parallelization count so that each job finishes in 10 minutes (ideally), based on the current average duration of test files.

The process is as follows:

  1. The rspec-foss-impact:pipeline-generate job depends on detect-tests and generate the CI config of the dynamic child pipeline
  2. The rspec-foss-impact:trigger job triggers the child pipeline using the CI config generated by rspec-foss-impact:pipeline-generate.
  • It depends on setup-test-env, retrieve-tests-metadata, compile-test-assets as-if-foss, and rspec-foss-impact:pipeline-generate.

Things to note

I've split the .gitlab-ci.yml file into dedicated files under .gitlab/ci/shared/ so that I can include these where needed in child pipeline config.

No behavior change is expected compared to the existing pipeline behavior.

Screenshots or screen recordings

Parent pipeline Trigger job Child pipeline
Screen_Shot_2022-09-14_at_11.40.54 Screen_Shot_2022-09-14_at_11.41.18 child-pipeline

How to set up and validate locally

Test case Job
No RSpec file to run https://gitlab.com/gitlab-org/gitlab/-/jobs/3074822872
Less than 2 nodes to run (no parallelization) https://gitlab.com/gitlab-org/gitlab/-/jobs/3074886292 => https://gitlab.com/gitlab-org/gitlab/-/pipelines/648926956
Maximum parallelization https://gitlab.com/gitlab-org/gitlab/-/jobs/3074315058 => https://gitlab.com/gitlab-org/gitlab/-/pipelines/648828499

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #372343 (closed).

Edited by Rémy Coutable

Merge request reports

Loading