Run reliable tests on GDK automatically
What does this MR do and why?
Runs reliable
tests automatically (allowed to fail) against GDK in CI in MR and master pipelines.
-
qa/specs/features/api/1_manage/rate_limits_spec.rb
doesn't behave as expected so it's excluded from GDK tests until it can be investigated. - Smoke test failures with load balancing enabled need investigation. They're now allowed to fail too.
Part of #388533 (closed) and gitlab-org/quality/quality-engineering&4 (closed)
Effect on pipelines
There should be no noticable difference in most cases. No change to the jobs that gdk-qa-reliable
is included in (i.e., code/CI/QA changes), but it will now always run instead of being manual. This means MR and master pipelines will have an extra 5 jobs that run in parallel, but they should start at the same time as gdk-qa-smoke
and run for about the same duration (give or take a few minutes).
Announcement (to be shared)
To be posted in Slack (#quality
, #development
, #engineering-fyi
) and the engineering WiR doc:
reliable
suite (https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/) against GDK in MR and master pipelines. The jobs are allowed to fail for now. We plan to make them block pipelines, but only after we're confident they do run reliably against GDK in CI.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.