Allow flaky test TestLimiter to fail in CI with warning (!1975) · Merge requests · GitLab.org / gitaly

Paul Okstad (ex-GitLab) requested to merge po-fix-flaky-test-limiter-too-high into master Mar 26, 2020

This change keeps our tech debt visible without disabling flaky tests in CI. Rather than ignore a test completely, this will add a new set of jobs to our pipeline that emits warnings when a flaky test fails. Any Go test that starts with "TestFlaky" will be run in these new jobs. Any test with the statement buildhelper.SkipFlakyTest(t) that meets criteria (i.e. starts with TestFlaky and the environment variable ENABLE_FLAKY_TESTS is set to "1") will be skipped in the regular test jobs.

Closes #2574 (closed)

Follow up issue: #2588 (closed)

Edited Mar 27, 2020 by Paul Okstad (ex-GitLab)

Allow flaky test TestLimiter to fail in CI with warning

Merge request reports