Skip to content

Exit immediately if running minimal reproduction fails

Peter Leitzen requested to merge pl-rspec-bisect-flaky into master

What does this MR do and why?

This MR ensures that scripts/rspec_bisect_flaky (added in !75457 (merged)) exits immediately if running a minimal reproduction fails.

Previously, we've run specs for all combinations and continued if a combination failed.

This MR also fixes some shellcheck offenses.

Screenshots or screen recordings

Shellcheck offenses

$ shellckeck scripts/rspec_bisect_flaky

In scripts/rspec_bisect_flaky line 13:
files=( $@ )
        ^-- SC2206: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.


In scripts/rspec_bisect_flaky line 25:
  bin/rspec $file $target
            ^---^ SC2086: Double quote to prevent globbing and word splitting.
                  ^-----^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean: 
  bin/rspec "$file" "$target"


In scripts/rspec_bisect_flaky line 29:
bin/rspec --bisect=verbose $@
                           ^-- SC2068: Double quote array expansions to avoid re-splitting elements.

For more information:
  https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
  https://www.shellcheck.net/wiki/SC2206 -- Quote to prevent word splitting/g...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

How to set up and validate locally

While trying to reproduce failures from #351431 (closed) I ran

$ scripts/rspec_bisect_flaky ee/spec/features/protected_branches_spec.rb ee/spec/features/groups/analytics/cycle_analytics/filters_and_data_spec.rb ee/spec/features/projects/pipelines/pipeline_spec.rb ee/spec/features/admin/subscriptions/admin_views_subscription_spec.rb ee/spec/features/ci_shared_runner_warnings_spec.rb ee/spec/features/issues/epic_in_issue_sidebar_spec.rb ee/spec/features/boards/boards_spec.rb ee/spec/features/admin/admin_dev_ops_reports_spec.rb ee/spec/features/projects/audit_events_spec.rb ee/spec/features/projects/mirror_spec.rb ee/spec/features/namespace_user_cap_reached_alert_spec.rb ee/spec/features/registrations/user_sees_new_onboarding_flow_spec.rb ee/spec/features/groups/seat_usage/seat_usage_spec.rb ee/spec/features/admin/licenses/show_user_count_threshold_spec.rb ee/spec/features/projects/members/invite_group_and_members_spec.rb ee/spec/features/groups/iterations/user_views_iteration_spec.rb ee/spec/features/epics/filtered_search/visual_tokens_spec.rb ee/spec/features/admin/admin_credentials_inventory_spec.rb ee/spec/features/groups/iterations/user_edits_iteration_spec.rb ee/spec/features/merge_request/user_sets_approval_rules_spec.rb ee/spec/features/ide/user_commits_changes_spec.rb ee/spec/features/groups/billing_spec.rb ee/spec/features/issues/issue_actions_spec.rb ee/spec/features/admin/groups/admin_changes_plan_spec.rb ee/spec/features/projects/insights_spec.rb ee/spec/features/projects/feature_flags/feature_flag_issues_spec.rb ee/spec/features/merge_requests/user_views_all_merge_requests_spec.rb ee/spec/features/groups/analytics/ci_cd_analytics_spec.rb ee/spec/features/projects/integrations/user_activates_jira_spec.rb ee/spec/features/registrations/combined_registration_spec.rb ee/spec/features/projects/quality/test_case_create_spec.rb ee/spec/features/admin/admin_interacts_with_push_rules_spec.rb ee/spec/features/groups/security/compliance_dashboards_spec.rb ee/spec/features/groups/iterations/user_edits_iteration_cadence_spec.rb ee/spec/features/dashboards/activity_spec.rb ee/spec/features/projects_spec.rb ee/spec/features/projects/feature_flags/user_deletes_feature_flag_spec.rb ee/spec/features/groups/new_spec.rb

I was surprised to see that the script kept running if after first failures appeared.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports

Loading