New script to find minimal order-dependent flaky specs
What does this MR do and why?
New script to find minimal order-dependent flaky specs
Contribute new script to quickly find minimal reproduction for
order-dependent flaky specs. It's much faster than rspec --bisect
.
Screenshots or screen recordings
Example run
$ cat a.txt
spec/lib/gitlab/relative_positioning/mover_spec.rb
spec/services/clusters/applications/uninstall_service_spec.rb
spec/services/clusters/gcp/verify_provision_status_service_spec.rb
spec/tasks/gitlab/cleanup_rake_spec.rb
cat a.txt | xargs scripts/rspec_bisect_flaky
$ cat a.txt | xargs scripts/rspec_bisect_flaky
+ for file in '"${files[@]}"'
+ bin/rspec spec/lib/gitlab/relative_positioning/mover_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
Test environment set up in 6.37978 seconds
......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Finished in 2 minutes 0 seconds (files took 10.8 seconds to load)
1190 examples, 0 failures
+ for file in '"${files[@]}"'
+ bin/rspec spec/services/clusters/applications/uninstall_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
Test environment set up in 6.214379 seconds
.....................
Finished in 19.45 seconds (files took 10.03 seconds to load)
21 examples, 0 failures
+ for file in '"${files[@]}"'
+ bin/rspec spec/services/clusters/gcp/verify_provision_status_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
Test environment set up in 6.216664 seconds
.....................
Finished in 23.77 seconds (files took 10.61 seconds to load)
21 examples, 0 failures
+ for file in '"${files[@]}"'
+ bin/rspec spec/tasks/gitlab/cleanup_rake_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
Test environment set up in 6.633838 seconds
...............
Finished in 14.53 seconds (files took 9.95 seconds to load)
15 examples, 0 failures
+ bin/rspec --bisect=verbose spec/lib/gitlab/relative_positioning/mover_spec.rb spec/services/clusters/applications/uninstall_service_spec.rb spec/services/clusters/gcp/verify_provision_status_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Bisect started using options: "spec/lib/gitlab/relative_positioning/mover_spec.rb spec/services/clusters/applications/uninstall_service_spec.rb spec/services/clusters/gcp/verify_provision_status_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb" and bisect runner: :fork
Running suite to find failures...
Bisect failed!
No failures found. Bisect only works in the presence of one or more failing examples.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Thong Kuah