RSpec: retrying only failures doesn't run any examples and Crystalball report is empty as a result
Job #2021655824 failed for b59c6630:
Trying to debug with !79329 (closed)
$ update_tests_mapping
/builds/gitlab-org/gitlab/tooling/lib/tooling/test_map_generator.rb:17:in `block in parse': undefined method `each' for nil:NilClass (NoMethodError)
from /builds/gitlab-org/gitlab/tooling/lib/tooling/test_map_generator.rb:13:in `each'
from /builds/gitlab-org/gitlab/tooling/lib/tooling/test_map_generator.rb:13:in `parse'
from scripts/generate-test-mapping:16:in `<main>'
Indeed, it seems some Crystalball files have no examples: https://gitlab.com/gitlab-org/gitlab/-/jobs/2023373567
Ok, so it’s related to https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148, we expect RSpec to rerun the failed examples but it seems to not run any (https://gitlab.com/gitlab-org/gitlab/-/jobs/2023373426):
Run options: include {:last_run_status=>"failed"}
All examples were filtered out
Test environment set up in 4.730209124 seconds
Finished in 8.19 seconds (files took 4 minutes 16.7 seconds to load)
0 examples, 0 failures
and the Crystalball report gets no examples.
This can be reproduced with the following file in spec/
rspec_last_run_results.txt and by running the following:
bin/rspec -Ispec -rspec_helper --color --format documentation --format RspecJunitFormatter --out rspec/junit_rspec-retry.xml --only-failures
Edited by Rémy Coutable