Run all backend specs in random order
Problem
In !93137 (merged) we started running all
To maintainer master
stability, all existing backend specs were excluded (via spec/support/rspec_order_todo.yml
) from running in random order so they still run in "defined" order (rspec --order defined
).
Proposed solution
To make spec files run in random order, check their order dependency with:
scripts/rspec_check_order_dependence <path> ...
scripts/rspec_check_order_dependence spec/models/project_spec.rb
scripts/rspec_check_order_dependence spec/models/project_spec.rb spec/models/project_label_spec.rb
scripts/rspec_check_order_dependence spec/models/project*
If the specs pass the check the script removes them from rspec_order_todo.yml
automatically.
If the specs fail the check they must be fixed before they can run in random order.
See https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-order
The list
As of 2023-04-19 the list spec/support/rspec_order_todo.yml
contains over 10.000 entries.
Implementation Guide
Any help is much appreciated
- Pick entries from
spec/support/rspec_order_todo.yml
- Check their order dependency via
scripts/rspec_check_order_dependence
- If specs
-
✅ Pass, commit the changes inspec/support/rspec_order_todo.yml
-
❌ Fail-
🅰 Fix the specs and go to step 2 -
🅱 Create an issue and ping@splattael
for triage
-
- Submit an MR and mention this issue via
Contributes to #407877
🎉
Example MRs
- !93143 (merged) - no changes needed
- !93187 (merged) - a few adjustments needed
Edited by Peter Leitzen