Skip to content

New script to find minimal order-dependent flaky specs

Thong Kuah requested to merge rspec_bisect_flaky into master

What does this MR do and why?

New script to find minimal order-dependent flaky specs

Contribute new script to quickly find minimal reproduction for order-dependent flaky specs. It's much faster than rspec --bisect.

Screenshots or screen recordings

Example run

$ cat a.txt 
spec/lib/gitlab/relative_positioning/mover_spec.rb
spec/services/clusters/applications/uninstall_service_spec.rb
spec/services/clusters/gcp/verify_provision_status_service_spec.rb
spec/tasks/gitlab/cleanup_rake_spec.rb
cat a.txt | xargs scripts/rspec_bisect_flaky
$ cat a.txt | xargs scripts/rspec_bisect_flaky 
+ for file in '"${files[@]}"'
+ bin/rspec spec/lib/gitlab/relative_positioning/mover_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Test environment set up in 6.37978 seconds
......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Finished in 2 minutes 0 seconds (files took 10.8 seconds to load)
1190 examples, 0 failures

+ for file in '"${files[@]}"'
+ bin/rspec spec/services/clusters/applications/uninstall_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Test environment set up in 6.214379 seconds
.....................

Finished in 19.45 seconds (files took 10.03 seconds to load)
21 examples, 0 failures

+ for file in '"${files[@]}"'
+ bin/rspec spec/services/clusters/gcp/verify_provision_status_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Test environment set up in 6.216664 seconds
.....................

Finished in 23.77 seconds (files took 10.61 seconds to load)
21 examples, 0 failures

+ for file in '"${files[@]}"'
+ bin/rspec spec/tasks/gitlab/cleanup_rake_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Test environment set up in 6.633838 seconds
...............

Finished in 14.53 seconds (files took 9.95 seconds to load)
15 examples, 0 failures

+ bin/rspec --bisect=verbose spec/lib/gitlab/relative_positioning/mover_spec.rb spec/services/clusters/applications/uninstall_service_spec.rb spec/services/clusters/gcp/verify_provision_status_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb
Bisect started using options: "spec/lib/gitlab/relative_positioning/mover_spec.rb spec/services/clusters/applications/uninstall_service_spec.rb spec/services/clusters/gcp/verify_provision_status_service_spec.rb spec/tasks/gitlab/cleanup_rake_spec.rb" and bisect runner: :fork
Running suite to find failures...
Bisect failed! 

No failures found. Bisect only works in the presence of one or more failing examples.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Thong Kuah

Merge request reports

Loading