Support fast-quarantining E2E tests
What does this MR do and why?
This adds support for fast-quarantining E2E examples. This depends on gitlab-qa!1168 (merged).
Related to gitlab-org/quality/quality-engineering/team-tasks#1777 (closed).
The example path should be given relatively to the top-level qa/
folder from where the E2E tests are run, e.g.
qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
Screenshots or screen recordings
https://gitlab.com/gitlab-org/gitlab/-/jobs/4379911523#L2162:
Create
Branch Rules Overview
adds a new branch rule (PENDING: Skipping ./qa/specs/features/browser_ui/3_create/repository/add_new_branch_rule_spec.rb[1:1:1] because it's been fast-quarantined.)
How to set up and validate locally
- Create a
rspec/fast_quarantine-gitlab.txt
file with the example identifiers that you'd like to quarantine, e.g.qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
- Run the test that's supposed to be quarantine, e.g.
cd qa && bundle exec rspec qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
- You should see an output saying that examples are skipped, e.g.
sets the default branch name for a new project (PENDING: Skipping ./qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb[1:1:1] because it's been fast-quarantined.)
[...]
Pending: (Failures listed here are expected and do not affect your suite's status)
1) Create Default branch name instance setting sets the default branch name for a new project
# Skipping ./qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb[1:1:1] because it's been fast-quarantined.
Failure/Error: raise SkipDeclaredInExample.new(message)
RSpec::Core::Pending::SkipDeclaredInExample
# /Users/remy/Code/GitLab/gdk/gitlab/spec/support/fast_quarantine.rb:20:in `block (2 levels) in <top (required)>'
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Rémy Coutable