Filter E2E Praefect tests to only run create tests
What does this MR do and why?
The praefect job provides a mechanism to run our E2E suite against an environment running praefect. However in the current implementation this suite runs the (almost) entire suite of tests. Many of these tests do not have any dependency on praefect/gitaly, nor even interact with the system except in an incidental manner (e.g. to create a project).
Instead - lets modify the job to run only the E2E tests defined in the Create stage. These tests are much more likely to focus on specific functionality that interacts with gitaly/praefect, and so can provide a good indication that the integration between rails and gitaly/praefect is stable, while also avoiding running an excessive number of tests that aren't adding value.
Results
Reducing number of tests
Taking a sample recent master job - https://gitlab.com/gitlab-org/gitlab/-/pipelines/899829163 and comparing to a pipeline in this MR https://gitlab.com/gitlab-org/gitlab/-/pipelines/899875768 we can observe results as follows.
| Before | Before Duration | After | After Duration |
|---|---|---|---|
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4472944320 | 37 minutes 57 seconds | https://gitlab.com/gitlab-org/gitlab/-/jobs/4473239435 | 21 minutes 43 seconds |
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4472944321 | 39 minutes 46 seconds | https://gitlab.com/gitlab-org/gitlab/-/jobs/4473239436 | 19 minutes 12 seconds |
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4472944322 | 38 minutes 3 seconds | https://gitlab.com/gitlab-org/gitlab/-/jobs/4473390178 | 18 minutes 33 seconds |
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4472944325 | 39 minutes 58 seconds | https://gitlab.com/gitlab-org/gitlab/-/jobs/4473239440 | 19 minutes 13 seconds |
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4472944329 | 41 minutes 42 seconds | https://gitlab.com/gitlab-org/gitlab/-/jobs/4473239441 | 20 minutes 1 second |
| Total Before | 3 hours 17 minutes 26 seconds | Total After | 1 hour 38 minutes 42 seconds |
Reducing number of jobs
By reducing the number of runners from 5->2 we can also save approx 30 minutes, as each job takes approx 10minutes to do initial setup work. With that accounted for, it seems like we can run this entire set of tests in approx 62 minutes.
| With 2 Runners | Duration |
|---|---|
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4479051851 | 31 minutes 31 seconds |
| https://gitlab.com/gitlab-org/gitlab/-/jobs/4479051854 | 30 minutes 23 seconds |
Overall outcome
- Reduction ~66% total duration for the praefect job
- Reducing number of runners for the job from 5->2
- Reduction of approx 2 hours 15 minutes from total suite execution time
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.