Skip to content

Run minimal rspec jobs based on test detection on Merge Requests pipelines

Albert Salim requested to merge caalberts-rspec-jobs-filtered into master

What does this MR do?

This MR creates rspec job variants that run minimal rspec tests and sets up rules to create minimal rspec jobs in MR pipelines in parallel to the full version. This is to allow experimenting with minimal rspec job and tracking the results, as part of approach outlined in #222369 (comment 457663853)

This MR builds on !48655 (merged), where detect-tests job is updated to detect tests based on Crystalball generated test mapping.

Approach

In this MR, we change the default RSpec jobs that run in a standard Merge Requests.

A standard MR pipeline consists of the following rspec jobs:

- rspec migration pg11
- rspec unit pg11
- rspec integration pg11
- rspec system pg11
- rspec-ee migration pg11
- rspec-ee unit pg11
- rspec-ee integration pg11
- rspec-ee system pg11
- rspec-ee unit pg11 geo
- rspec-ee integration pg11 geo
- rspec-ee system pg11 geo
- rspec migration pg11 as-if-foss
- rspec unit pg11 as-if-foss
- rspec integration pg11 as-if-foss
- rspec system pg11 as-if-foss

Rule changes

Before this MR:

job name master, scheduled pipelines, MR that changes CI on MR pipelines on MR RUN AS-IF-FOSS on MR RUN ALL RSPEC
rspec * pg11 ✅ ✅ ✅ ✅
rspec-ee * pg11 ✅ ✅ ✅ ✅
rspec-ee * pg11 geo ✅ ✅ ✅ ✅
rspec-ee * pg11 as-if-foss 🚫 🚫 ✅ 🚫

With changes in this MR:

job name master, scheduled pipelines, MR that changes CI on MR pipelines on MR RUN AS-IF-FOSS on MR RUN ALL RSPEC
rspec * pg11 ✅ ✅ ✅ ✅
rspec * pg11 minimal ⚠ 🚫 ✅ ✅ 🚫
rspec-ee * pg11 ✅ ✅ ✅ ✅
rspec-ee * pg11 minimal ⚠ 🚫 ✅ ✅ 🚫
rspec-ee * pg11 geo ✅ ✅ ✅ ✅
rspec-ee * pg11 geo minimal ⚠ 🚫 ✅ ✅ 🚫
rspec-ee * pg11 as-if-foss 🚫 🚫 ✅ 🚫
rspec-ee * pg11 as-if-foss minimal ⚠ 🚫 🚫 ✅ 🚫
  • ✅ => job is created
  • 🚫 => job is not created
  • ⚠ => new rule

By running as separate jobs, we could run both minimal and full job in parallel to compare the results if needed.

RSpec jobs that are dependent on MR title keywords such as RUN ALL RSPEC are not changed.

Feature Toggle

The rspec minimal jobs will only be created if $RSPEC_MINIMAL_ENABLED="true" is set in the pipeline.

Post-experiment

After experimentation is completed, if we wish to completely switch to minimal rspec job in MRs, we'd need to follow up with an MR to:

  1. Change rules to not run full job if a merge request title doesn't contains RUN ALL RSPEC or doesn't change CI
job name master, scheduled pipelines, MR that changes CI on MR pipelines on MR RUN AS-IF-FOSS on MR RUN ALL RSPEC
rspec * pg11 ✅ 🚫 ⚠ 🚫 ⚠ ✅
rspec * pg11 minimal 🚫 ✅ ✅ 🚫
rspec-ee * pg11 ✅ 🚫 ⚠ 🚫 ⚠ ✅
rspec-ee * pg11 minimal 🚫 ✅ ✅ 🚫
rspec-ee * pg11 geo ✅ 🚫 ⚠ 🚫 ⚠ ✅
rspec-ee * pg11 geo minimal 🚫 ✅ ✅ 🚫
rspec-ee * pg11 as-if-foss 🚫 🚫 ⚠ 🚫 ⚠ 🚫
rspec-ee * pg11 as-if-foss minimal 🚫 🚫 ✅ 🚫
  • ✅ => job is created
  • 🚫 => job is not created
  • ⚠ => rule changed

Part of #263407 (closed) #222369 (closed) #270401 (closed)

Verifications

Edited by Albert Salim

Merge request reports