Skip to content

Make flaky test regexp stricter

Adam Cohen requested to merge make-flaky-test-regexp-more-strict into master

What does this MR do and why?

Pass pipeline with warnings when failed test is... (!104332 - merged) added logic for passing pipelines with warnings if flaky tests were present, however, the regexp used in that MR is too permissive and resulted in false positives. For example:

This job contains a flaky test:

1) Member when activating a member when user cap is reached when user has another awaiting membership keeps the member awaiting
   Failure/Error: connection.send(...)
   ActiveRecord::InvalidForeignKey:
     PG::ForeignKeyViolation: ERROR:  insert or update on table "members" violates foreign key constraint "fk_2e88fb7ce9"
     DETAIL:  Key (user_id)=(1242) is not present in table "users".

   # ./ee/spec/models/member_spec.rb:397:in `block (5 levels) in <top (required)>'

A test was flaky and succeeded after being retried. Checking to see if flaky test is part of this MR...
Searching flaky tests for spec/helpers/todos_helper_spec.rb
Searching flaky tests for spec/models/member_spec.rb
Exiting with code 137 because the flaky test was part of this MR.

This is a false positive because it matched the file changed in the MR spec/models/member_spec.rb with the flaky test ./ee/spec/models/member_spec.rb due to a regex that's too permissive.

This MR makes the regexp stricter in order to eliminate these false positives.

What are the relevant issue numbers?

Pass pipeline with warnings when failed rspec j... (#383810 - closed)

Testing

Results of two different successfully retried tests:

  • Test was part of the MR that triggered the pipeline:

    A test was flaky and succeeded after being retried. Checking to see if flaky test is part of this MR...
    Flaky test '^\./ee/spec/graphql/types/vulnerability_type_spec.rb' was found in the list of files changed by this MR.
    Exiting with code 137.
    
    ERROR: Job failed: exit code 137

    Result: job passes with warnings.

  • Test was not part of the MR that triggered the pipeline:

    A test was flaky and succeeded after being retried. Checking to see if flaky test is part of this MR...
    Flaky test was not part of this MR.
    
    Job succeeded

    Result: job passes without warnings.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Adam Cohen

Merge request reports

Loading