Skip to content

ci: Fix the retrying of failing specs for `-ee` jobs

Rémy Coutable requested to merge 351341-rspec-retrying-only-failures-doesn-t-run-any-examples-and-crystalball-report-is-empty-as-a into master

What does this MR do and why?

The "only-failure" feature takes the intersection of failed examples
with the list of files that would run without the `--only-failures`
flag.
In our case, the problem only appeared for `-ee` jobs since the default
RSpec pattern is `spec/**/*_spec.rb`, the intersection would be empty.
Passing the pattern from the first run makes the intersection return
the "failed" examples correctly.

Also, disable Crystalball for the retry run so that the first run report
is used.

How to set up and validate locally

  1. Put the attached file to rspec/rspec_last_run_results.txt. This is a first RSpec run with two failed EE examples, from https://gitlab.com/gitlab-org/gitlab/-/jobs/2021655824.
  2. Run the following command (reproducing the bugged behavior): 
    RSPEC_LAST_RUN_RESULTS_FILE="rspec/rspec_last_run_results.txt" be rspec -Ispec -rspec_helper --color --format documentation --format RspecJunitFormatter --out rspec/junit_rspec-retry.xml --only-failures
  3. No examples should run.
  4. Now, run the following command (reproducing the fixed behavior): 
    RSPEC_LAST_RUN_RESULTS_FILE="rspec/rspec_last_run_results.txt" be rspec -Ispec -rspec_helper --color --format documentation --format RspecJunitFormatter --out rspec/junit_rspec-retry.xml --only-failures --pattern "{ee/}spec/{commands,controllers,mailers,requests}{,/**/}*_spec.rb"
  5. Two examples should run and pass.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #351341 (closed).

rspec_last_run_results.txt

Edited by Rémy Coutable

Merge request reports