Skip to content

Support fast-quarantining E2E tests

Rémy Coutable requested to merge support-fast-quarantining-e2e-tests into master

What does this MR do and why?

This adds support for fast-quarantining E2E examples. This depends on gitlab-qa!1168 (merged).

Related to gitlab-org/quality/quality-engineering/team-tasks#1777 (closed).

The example path should be given relatively to the top-level qa/ folder from where the E2E tests are run, e.g.

qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14

Screenshots or screen recordings

https://gitlab.com/gitlab-org/gitlab/-/jobs/4379911523#L2162:

Create
  Branch Rules Overview
    adds a new branch rule (PENDING: Skipping ./qa/specs/features/browser_ui/3_create/repository/add_new_branch_rule_spec.rb[1:1:1] because it's been fast-quarantined.)

How to set up and validate locally

  1. Create a rspec/fast_quarantine-gitlab.txt file with the example identifiers that you'd like to quarantine, e.g. qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
  2. Run the test that's supposed to be quarantine, e.g. cd qa && bundle exec rspec qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
  3. You should see an output saying that examples are skipped, e.g.
    sets the default branch name for a new project (PENDING: Skipping ./qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb[1:1:1] because it's been fast-quarantined.)

[...]

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Create Default branch name instance setting sets the default branch name for a new project
     # Skipping ./qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb[1:1:1] because it's been fast-quarantined.
     Failure/Error: raise SkipDeclaredInExample.new(message)
       RSpec::Core::Pending::SkipDeclaredInExample
     # /Users/remy/Code/GitLab/gdk/gitlab/spec/support/fast_quarantine.rb:20:in `block (2 levels) in <top (required)>'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports

Loading