Run specs in random order multiple times
What does this MR do and why?
Sometimes, running specs in random order only once is not sufficient to catch flaky specs.
This MR also fixes running specs in random order with spring when enabled.
Refs !157178 (comment 1968533439)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
scripts/rspec_check_order_dependence ./ee/spec/config/metrics/every_metric_definition_spec.rb
# See spec being run 7 times:
# - 1x defined order
# - 1x reverse order
# - 5x random order
Running in defined order:
Running in reverse order:
Running in random order (1/5):
Running in random order (2/5):
Running in random order (3/5):
Running in random order (4/5):
Running in random order (5/5):
export ENABLE_SPRING=1
bin/rspec ./ee/spec/config/metrics/every_metric_definition_spec.rb
bin/rspec ./ee/spec/config/metrics/every_metric_definition_spec.rb
# Seed is now different
bin/rspec --seed 42 ./ee/spec/config/metrics/every_metric_definition_spec.rb
# Seed is now pinned
Edited by Peter Leitzen