Skip to content

Automate verify release deploy for monthly release

Mawreen Dela Cruz requested to merge madelacruz/monthly-verify-release-deploy into master

What does this MR do and why?

Describe in detail what your merge request does and why.

Similar to the MR which adds a job to the patch release pipeline to validate if the version is deployed to release.gitlab.net, this MR adds a similar job to the monthly release pipeline.

Related issue: gitlab-com/gl-infra/delivery#20449 (closed)

Testing

Pipeline creation

This was tested in this repo (gitlab-org/release-tools), just to see that the pipeline gets created as we expect it.

With SECURITY_RELEASE_PIPELINE=true

With SECURITY_RELEASE_PIPELINE='tag'

image.png

image.png

This verification gets created with the other tagging jobs, and should run at the end.

Pipeline

Pipeline

Configuration testing

This was to test the configuration of the pipeline (the job order).

Step Screenshot

For testing - Echo a log line instead of actually doing sth (Testing commit)

  • Tag jobs were skipped with echos
  • The wait time was deduced to 2m instead of 10m

Initialized as manual pipeline (Pipeline)

  • SLACK_TEST = true
  • MONTHLY_RELEASE_PIPELINE = tag_day

image.png

After monthly_release_tag_day:prepare is manually triggered

image.png
Slack notification to signal start of tag process, as per usual

image.png

monthly_release_tag_day:check_omnibus_packages_tagging stage jobs are finished

  • Waits for 2 minutes

image.png

`monthly_release_tag_day:verify_release_deploy stage job is started and successfully completed

image.png

Execution Testing

This test is simply to test the execution of the verify_release_deploy job. Mainly we're looking for the slack notifications on success and failure verification cases.

Step Screenshot

For testing - mimic a passing validation (Testing commit, update)

  • Removed the monthly_release:tag_day stage jobs
  • Removed the delay

image.png

This is the version that's going to be returned as the expected monthly release.

Since today's not the tagging day, this would normally fail, so to simulate the day of the tagging, we hardcoded the version to be the same as the version deployed to the release environment.

Same as above, set the pipeline vars for testing (Pipeline)

  • SLACK_TEST = true
  • MONTHLY_RELEASE_PIPELINE = tag

image.png

The pipeline for testing is much shorter, since we're not re-testing the security_release:tag stage jobs.
Slack notification

image.png

The job successfully completed, and posts a successful execution message to Slack.

image.png

Note: Failure and retries are tested in this MR for the deploy validator of the patch release version: !3417 (merged)

Tag Day Monthly Release Issue - sample

Tag day: Wednesday, Sep 18

  • Confirm that final RC version has passed automated tests
    • Ensure tests are green on [CNG]
  • Start the monthly_release_tag_day:prepare job in the monthly release pipeline: https://example.com/foo/bar/-/pipelines/1``
    • Jobs to verify the integrity of the stable branches (monthly_release_tag_day:ensure_stable_branches_green) and the mirror status (monthly_release_tag_day:mirror_status) will start automatically.
  • Once the above jobs are green, trigger the monthly_release_tag_day:start_tag job: https://example.com/foo/bar/-/pipelines/1``
    • This job will tag the final 17.4.0 version.
  • After tagging, wait for some time (around 1.5 hours), and then start monthly_release_tag_day:check_omnibus_packages_tagging job in the monthly release pipeline: https://example.com/foo/bar/-/pipelines/1`` to check if the packages are built. If the job fails, check and fix the issue, and rerun the job.``
    • Check that the CNG Images are built: CNG builds.
    • Check that the [Helm] charts for 17.4.0 are built. Note that the Helm chart version is 9 major versions behind GitLab version. For example, GitLab 17.3 has the corresponding Helm chart 8.3, so the stable branch is 8-3-stable.
    • Verify that release.gitlab.net is running the latest patch version by either:
  • Validate 17.4.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

Content

  • Automate verify release deploy for monthly release

Author Check-list

  • Has documentation been updated?
Edited by Mawreen Dela Cruz

Merge request reports