Create loop_runner module to run test repeatedly
What does this MR do?
- This MR creates a new type of specs runner that will repeatedly run an E2E test for a specified period
- The period can be set using an Environment variable "GITLAB_QA_LOOP_RUNNER_MINUTES". The default is 1 minute.
- This Loop Runner will be called instead of Runner when the shared_attribute '--loop' is included as a CLI option, e.g.
bin/qa Test::Instance::All http://localhost:3000 --loop qa/specs/features/browser_ui/1_manage/project/create_project_spec.rbWhy do we want to do this?
We want a way to run certain E2E tests repeatedly (e.g. for 40 minutes) as part of a CI job.
We want to check that zero-downtime upgrades of GitLab are truly zero-downtime.
Why do it this way?
Previous idea was to loop over the test example, instead of over the call to the runner as done here. That method involved re-organizing the spec (using shared_examples and an rspec tag), which would have to be done for any test we'd want to loop over (not ideal). Also, the logic for how often to run the test doesn't seem to belong in the spec file itself.
This method seems less intrusive, but comes with its own questions (see next).
Questions
-
Is there a more boring solution I'm overlooking here? -
Is this a 'better' way than looping over the test? Is it less desirable to repeatedly call RSpec::Core::Runner to run one test, than to call the Runner once and tell it to run the test multiple times? -
~~When I test this using the login spec (a relatively quick running test), after ~2-3 successful runs I get a "Retry later" message. Why does this happen...is there some type of throttling, do we need to 'wait' between attempts?~~
-
The rspec-retry gem's 'retry' option seems to get triggered during repeat runs, even if the previous run was successful:
1. First run: 1 example
1. Second run: 2 examples (2nd retry, then a 'non-retry' run)
1. Third run: 3 examples (2nd retry, 3rd retry, then a 'non-retry' run)
Right now I'm not clear why or if this is a big issue.