Skip to content

ci: Allow to fast-quarantine RSpec examples

Rémy Coutable requested to merge 204-create-a-flaky-tests-ssot-outside-of-gitlab-org-gitlab into master

What does this MR do and why?

This uses an external source of truth at https://gitlab-org.gitlab.io/quality/engineering-productivity/fast-quarantine/rspec/fast_quarantine-gitlab.txt so that we'll be able to quickly & temporarily quarantine problematic tests without having to open/merge a merge request.

That could be especially useful for Delivery pipelines. We’d need to support E2E tests fast-quarantining, though, as these are the ones that usually need to be quarantined for Delivery pipelines.

Examples

With https://gitlab-org.gitlab.io/quality/engineering-productivity/fast-quarantine/rspec/fast_quarantine-gitlab.txt having the following content:

./spec/tasks/gitlab/usage_data_rake_spec.rb[1:5:2:1]
ee/spec/features/boards/swimlanes/epics_swimlanes_sidebar_spec.rb

Example of a quarantined example via example id:

Skipping ./spec/tasks/gitlab/usage_data_rake_spec.rb[1:5:2:1] 'gitlab:usage data take tasks generate_and_send generate_ci_template_events generates /builds/gitlab-org/gitlab/lib/gitlab/usage_data_counters/known_events/ci_templates.yml' because it's been fast-quarantined with 'spec/tasks/gitlab/usage_data_rake_spec.rb[1:5:2:1]'.

Example of a quarantined file:

epics swimlanes sidebar
Starting the Capybara driver server...
  in project boards
    behaves like issue boards sidebar
Skipping ./ee/spec/features/boards/swimlanes/epics_swimlanes_sidebar_spec.rb[1:1:1:1] 'epics swimlanes sidebar in project boards behaves like issue boards sidebar shows sidebar when clicking issue' because it's been fast-quarantined with 'ee/spec/features/boards/swimlanes/epics_swimlanes_sidebar_spec.rb'.
      shows sidebar when clicking issue (PENDING: around hook at ./spec/support/fast_quarantine.rb:27 did not execute the example)
Skipping ./ee/spec/features/boards/swimlanes/epics_swimlanes_sidebar_spec.rb[1:1:1:2] 'epics swimlanes sidebar in project boards behaves like issue boards sidebar closes sidebar when clicking issue' because it's been fast-quarantined with 'ee/spec/features/boards/swimlanes/epics_swimlanes_sidebar_spec.rb'.
[...]

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to gitlab-org/quality/engineering-productivity/team#204 (closed).

Edited by Rémy Coutable

Merge request reports