Make flaky test regexp stricter
What does this MR do and why?
Pass pipeline with warnings when failed test is... (!104332 - merged) added logic for passing pipelines with warnings if flaky tests were present, however, the regexp used in that MR is too permissive and resulted in false positives. For example:
This job contains a flaky test:
1) Member when activating a member when user cap is reached when user has another awaiting membership keeps the member awaiting
Failure/Error: connection.send(...)
ActiveRecord::InvalidForeignKey:
PG::ForeignKeyViolation: ERROR: insert or update on table "members" violates foreign key constraint "fk_2e88fb7ce9"
DETAIL: Key (user_id)=(1242) is not present in table "users".
# ./ee/spec/models/member_spec.rb:397:in `block (5 levels) in <top (required)>'
A test was flaky and succeeded after being retried. Checking to see if flaky test is part of this MR...
Searching flaky tests for spec/helpers/todos_helper_spec.rb
Searching flaky tests for spec/models/member_spec.rb
Exiting with code 137 because the flaky test was part of this MR.
This is a false positive because it matched the file changed in the MR spec/models/member_spec.rb
with the flaky test ./ee/spec/models/member_spec.rb
due to a regex that's too permissive.
This MR makes the regexp stricter in order to eliminate these false positives.
What are the relevant issue numbers?
Pass pipeline with warnings when failed rspec j... (#383810 - closed)
Testing
Results of two different successfully retried tests:
-
Test was part of the MR that triggered the pipeline:
A test was flaky and succeeded after being retried. Checking to see if flaky test is part of this MR... Flaky test '^\./ee/spec/graphql/types/vulnerability_type_spec.rb' was found in the list of files changed by this MR. Exiting with code 137. ERROR: Job failed: exit code 137
Result: job passes with warnings.
-
Test was not part of the MR that triggered the pipeline:
A test was flaky and succeeded after being retried. Checking to see if flaky test is part of this MR... Flaky test was not part of this MR. Job succeeded
Result: job passes without warnings.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.