Integrate orchestrated e2e tests in to gitlab project pipeline
What does this MR do and why?
Describe in detail what your merge request does and why.
Implements package-and-qa
as a child pipeline: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1334+
Pros:
- Moves test execution to where the code is
- Easier to see the test results, integrates test results in to merge request widgets once #363302 is done
- Cancelling redundant child pipeline should work correctly once cascading cancel is implemented
- Should allow to simplify rules and pipeline setup in
gitlab-qa
project since it has to handle pipelines for several upstream triggers - Static yml definition makes it easier to read which jobs are included in the pipeline and simplifies maintenance
- Improves feature toggle state change testing, omnibus image is only built once
- Adds update-minor and update-major test jobs
Cons:
-
gitlab-qa
now servers as a single point for running all e2e orchestrated tests, now this wouldn't be the case - Making reusable templates and extracting to
pipeline-common
might be difficult and might increase complexity and maintenance
Some notable changes compared to gitlab-qa
project pipeline implementation:
- Replaces somewhat ambiguous term
package-and-qa
withpackage-and-test
and jobs are prefixed withe2e
to make it clear what it does - Removes very verbose duplication of
quarantine
type jobs for every single job type in favour of one single parametrized manualquarantine
job - More reuse of templates in
pipeline-common
project - Selective test execution is done via pipeline rules rather than dynamically generating whole pipeline yml
- Moves all helpers in to the
qa
folder for better concern separation - Introduces new type of upgrade jobs which test upgrade from previous minor and major gitlab versions
- Properly locks
gitlab-qa
version viaGemfile.lock
Tasks:
-
Add ci runner for e2e test execution -
Add all of the missing job types -
Set up all necessary environment variables -
Set up issue creation/linking scripts -
Set up knapsack report regeneration
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Test Runs
- FF updated, run
ee:instance-parallel-ff-inverse
andee:instance-parallel
: https://gitlab.com/gitlab-org/gitlab/-/pipelines/621416820 - FF updated + QA framework change, run full suite +
ee:instance-ff-inverse
: https://gitlab.com/gitlab-org/gitlab/-/pipelines/621524316 - FF deleted, run only
ee:instance-parallel
: https://gitlab.com/gitlab-org/gitlab/-/pipelines/622607964 - Single spec change + stage label, specific spec executed: https://gitlab.com/gitlab-org/gitlab/-/pipelines/621887497
- Framework changes + stage label, all specs executed: https://gitlab.com/gitlab-org/gitlab/-/pipelines/622522601
- Quarantine only change, empty no-op pipeline: https://gitlab.com/gitlab-org/gitlab/-/jobs/2933298569
- BE change + stage label, only stage specific specs executed: https://gitlab.com/gitlab-org/gitlab/-/pipelines/622170898
Issues
Closes: package-and-qa job does not auto run based on M... (#366871 - closed)
Closes: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1334+
Closes: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1415+
Closes: Create a new orchestrated scenario to upgrade f... (gitlab-qa#672 - closed)
Closes: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1417+