[CI] QA only tests against latest version from CHART_VERSIONS
Summary
At the moment, only the latest version from CHART_VERSIONS is created as a review app and QA tested. The second and third versions are not deployed or QA tested (only unit tested).
Current behavior
Only the latest chart version is deployed and QA tested.
Expected behavior
All three chart versions are deployed and QA tested.
Considerations
- This will triple the CI load, so we'd need to ensure we have enough capacity to do this. What helps is that we already make the review environment stages
manual
, so they won't be running on every MR commit, which allows us to control when cluster resources are utilized. - In gitlab-org/distribution/team-tasks#1413, we're looking into stopping certain tests on
master
andstable
branches. This could also be applied to the topic here, since a lot of the QA tests that run here have already been run in the Charts CI. It's not a 1:1 reflection though, since the Operator does some manipulation of the objects from the charts, so it's still worth testing to some degree - but potentially in a separate or downstream pipeline. - We considered doing this in a scheduled/nightly pipeline, but were concerned that this would cause errors to drift from their source MRs and require manually tracing them back.
Acceptance criteria
-
A decision is made whether or not to deploy and test all 3 CHART_VERSIONS -
If decision is yes
, then all 3 CHART_VERSIONS are deployed and QA tested in CI -
MR template is created with a reminder to run a full QA pipeline prior to merge(this is now a project setting which requires a successful pipeline run prior to merge)
Edited by Mitchell Nielsen