Add release.gitlab.net deployment validator for patch release pipeline

What does this MR do and why?

After we tag the versions during a patch release, the latest version package gets deployed to release.gitlab.net.

Currently, the release manager has a manual task on the patch release task issue to check the version deployed to release.gitlab.net/help.

This will automate that process, so that this validator will run 2 hours and 30 minutes after the versions have been tagged.

(I was looking at past release timelines for an appropriate wait time, but we've had a few hiccups in the past few patch releases during this phase, so I'm not sure if that's too short/long in actuality.)

Addresses: gitlab-com/gl-infra/delivery#20235 (closed)

Content

• Add CI job to call validator awhile after tagging
• Add release deployment validator for patch release

Testing

Pipeline creation

This was tested in this repo (gitlab-org/release-tools), just to see that the pipeline gets created as we expect it.

With SECURITY_RELEASE_PIPELINE=true	With SECURITY_RELEASE_PIPELINE='tag'
	This verification gets created with the other tagging jobs, and should run at the end.
Pipeline	Pipeline

Configuration testing

This was to test the configuration of the pipeline (the job order).

Step	Screenshot
For testing - Echo a log line instead of actually doing sth (Testing commit) Tag jobs were skipped with echos The wait time was deduced to 5m instead of 210m
Initialized as manual pipeline (Pipeline) IS_TESTING = true SECURITY_RELEASE_PIPELINE = tag
After security_release:tag:start is manually triggered
Slack notification to signal start of tag process, as per usual
security_release:tag stage jobs are finished Waits for 5 minutes
security_release:verify_release_deploy stage job is started and successfully completed

Execution Testing

This test is simply to test the execution of the verify_release_deploy job. Mainly we're looking for the slack notifications on success and failure verification cases.

Step	Screenshot
For testing - mimic a passing validation (Testing commit) Removed the security_release:tag stage jobs Removed the delay	This is the version that's going to be returned as the latest patch release. On the day of the tagging, this would actually be the first of the to-be-tagged versions. Since today's not the tagging day, this would normally fail, so to simulate the day of the tagging, we hardcoded the version to be the same as the version deployed to the release environment.
Same as above, set the pipeline vars for testing (Pipeline) IS_TESTING = true SECURITY_RELEASE_PIPELINE = tag	The pipeline for testing is much shorter, since we're not re-testing the security_release:tag stage jobs.
Slack notification
The job successfully completed, and posts a successful execution message to slack.
Now let's test the failure case by changing the hardcoding the returned version to "17.3.0" (commit)
Same as above, create a new testing pipeline (Pipeline)
The slack notification to signal the start is the same as above.
The job "successfully" fails, and here is the slack failure notification

Test retrying

• Test the failure case by changing the hardcoding the returned version to "17.3.0" (commit)

• Then execute:

  [1] pry(main)> ReleaseTools::Security::ReleaseDeployValidator.new.execute
  2024-08-27 09:10:41.011937 D [security] ReleaseTools::GitlabReleaseClient -- [HTTParty] [2024-08-27 09:10:41 +0200] 200 "GET https://release.gitlab.net/api/v4/version" - 
  2024-08-27 09:10:41.012551 W [security] ReleaseTools::Security::ReleaseDeployValidator -- Release.gitlab.net is not running the latest patch version. Retrying.
  2024-08-27 09:11:41.327700 D [security] ReleaseTools::GitlabReleaseClient -- [HTTParty] [2024-08-27 09:11:41 +0200] 200 "GET https://release.gitlab.net/api/v4/version" - 
  2024-08-27 09:11:41.327805 W [security] ReleaseTools::Security::ReleaseDeployValidator -- Release.gitlab.net is not running the latest patch version. Retrying.

As you can see, the two retries are one minute away from each other.