Release preparation extended maintenance policy (!2098) · Merge requests · GitLab.org / release-tools

Steve Abrams requested to merge delivery2731-versionless-patch-prepare into master Dec 19, 2022

🌲 What does this MR do and why?

Updates the release:prepare task to accept a nil version argument. When no argument is given (/chatops release prepare):

The new blog post is generated
An issue is opened and assigned to release managers using a new template. The template is blank in this MR and will be added in gitlab-com/gl-infra/delivery#2655 (closed).

This MR will be followed by gitlab-com/chatops!357 (merged), which modifies chatops to allow for no version argument to be given.

📽 Screenshots

Extended maintenance policy patch (no version is given) release:prepare

In this example, I uncommented the content in templates/patch_multi_version.md.erb so that it would be logged to the terminal output.

The requests that happen after are all related to the blog post generation.


~/workspace/gitlab-org/release-tools (delivery2731-versionless-patch-prepare ✔) TEST='test' RELEASE_BOT_PRODUCTION_TOKEN=$TOKEN RELEASE_BOT_OPS_TOKEN=$OPS_TOKEN RELEASE_BOT_VERSION_TOKEN=test be rake release:prepare

# Releases 15.7.1, 15.6.4, 15.5.7

To be implemented in https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2732
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:54.398337 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:54 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fbuild%2FCNG/repository/compare" -
2022-12-23 09:44:54.399200 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/build/CNG, :tag=>"v15.7.0", :branch=>"15-7-stable", :commits=>0}
2022-12-23 09:44:54.399370 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/build/CNG, :version=>"15.7.0", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:54.821937 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:54 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitaly/repository/compare" -
2022-12-23 09:44:54.822579 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitaly, :tag=>"v15.7.0", :branch=>"15-7-stable", :commits=>0}
2022-12-23 09:44:54.822634 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitaly, :version=>"15.7.0", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:55.317599 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:55 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/repository/compare" -
2022-12-23 09:44:55.319909 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitlab, :tag=>"v15.7.0-ee", :branch=>"15-7-stable-ee", :commits=>0}
2022-12-23 09:44:55.319967 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitlab, :version=>"15.7.0", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:55.615853 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:55 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-pages/repository/compare" -
2022-12-23 09:44:55.616068 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitlab-pages, :tag=>"v15.7.0", :branch=>"15-7-stable", :commits=>0}
2022-12-23 09:44:55.616092 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitlab-pages, :version=>"15.7.0", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:56.039813 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:56 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fomnibus-gitlab/repository/compare" -
2022-12-23 09:44:56.040063 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/omnibus-gitlab, :tag=>"15.7.0+ee.0", :branch=>"15-7-stable", :commits=>0}
2022-12-23 09:44:56.040084 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/omnibus-gitlab, :version=>"15.7.0", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:56.407067 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:56 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fbuild%2FCNG/repository/compare" -
2022-12-23 09:44:56.407714 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/build/CNG, :tag=>"v15.6.3", :branch=>"15-6-stable", :commits=>0}
2022-12-23 09:44:56.407772 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/build/CNG, :version=>"15.6.3", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:56.751590 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:56 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitaly/repository/compare" -
2022-12-23 09:44:56.752201 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitaly, :tag=>"v15.6.3", :branch=>"15-6-stable", :commits=>0}
2022-12-23 09:44:56.752251 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitaly, :version=>"15.6.3", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:57.159243 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:57 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/repository/compare" -
2022-12-23 09:44:57.159442 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitlab, :tag=>"v15.6.3-ee", :branch=>"15-6-stable-ee", :commits=>0}
2022-12-23 09:44:57.159461 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitlab, :version=>"15.6.3", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:57.556250 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:57 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-pages/repository/compare" -
2022-12-23 09:44:57.557134 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitlab-pages, :tag=>"v15.6.3", :branch=>"15-6-stable", :commits=>0}
2022-12-23 09:44:57.557796 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitlab-pages, :version=>"15.6.3", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:57.900891 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:57 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fomnibus-gitlab/repository/compare" -
2022-12-23 09:44:57.902673 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/omnibus-gitlab, :tag=>"15.6.3+ee.0", :branch=>"15-6-stable", :commits=>0}
2022-12-23 09:44:57.902729 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/omnibus-gitlab, :version=>"15.6.3", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:58.388049 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:58 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fbuild%2FCNG/repository/compare" -
2022-12-23 09:44:58.389065 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/build/CNG, :tag=>"v15.5.6", :branch=>"15-5-stable", :commits=>0}
2022-12-23 09:44:58.389110 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/build/CNG, :version=>"15.5.6", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:58.949300 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:58 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitaly/repository/compare" -
2022-12-23 09:44:58.952407 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitaly, :tag=>"v15.5.6", :branch=>"15-5-stable", :commits=>1}
2022-12-23 09:44:59.595631 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:59 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitaly/repository/commits/e51f6b02d702e02a06166043e531a25949a875bb/merge_requests" -
2022-12-23 09:44:59.596035 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitaly, :version=>"15.5.6", :count=>1}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:44:59.953992 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:44:59 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/repository/compare" -
2022-12-23 09:44:59.954772 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitlab, :tag=>"v15.5.6-ee", :branch=>"15-5-stable-ee", :commits=>1}
2022-12-23 09:45:00.523003 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:45:00 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/repository/commits/8763f05c3ba4a57a3b9a3b180945701c296865b2/merge_requests" -
2022-12-23 09:45:00.524015 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitlab, :version=>"15.5.6", :count=>1}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.
2022-12-23 09:45:00.864564 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:45:00 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-pages/repository/compare" -
2022-12-23 09:45:00.865131 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/gitlab-pages, :tag=>"v15.5.6", :branch=>"15-5-stable", :commits=>0}
2022-12-23 09:45:00.865183 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/gitlab-pages, :version=>"15.5.6", :count=>0}
WARNING: Please convert ObjectifiedHash object to hash before calling Hash methods on it.

# Draft: Adding 15.5.1 blog post

2022-12-23 09:45:01.357071 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-23 09:45:01 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Fomnibus-gitlab/repository/compare" -
2022-12-23 09:45:01.357266 I [dry-run] ReleaseTools::PatchRelease::UnreleasedCommits -- Fetching unreleased commits -- {:project=>gitlab-org/omnibus-gitlab, :tag=>"15.5.6+ee.0", :branch=>"15-5-stable", :commits=>0}
2022-12-23 09:45:01.357288 I [dry-run] ReleaseTools::PatchRelease::UnreleasedMergeRequests -- Fetched unreleased merge requests -- {:project=>gitlab-org/omnibus-gitlab, :version=>"15.5.6", :count=>0}
Add blog post for 15.5.1 patch release.

Patch issue: test.gitlab.com

Single patch (no change) release:prepare[15.7.0]

Succeeds and uses the patch.md.erb template as expected


~/workspace/gitlab-org/release-tools (delivery2731-versionless-patch-prepare ✗) TEST='test' RELEASE_BOT_PRODUCTION_TOKEN=$TOKEN RELEASE_BOT_OPS_TOKEN=$OPS_TOKEN be rake release:prepare\[15.7.0\]

# Release 15.7



## First steps

- [ ] Change `#f_upcoming_release` topic with `/topic 15.7.0: `
- [ ] [Check whether the auto-deploy branch schedule] fits the Release Managers' working hours. Adjust if needed and make sure any changes are updated in [the handbook](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/#gitlabcom-deployments-process).
- [ ] Consider [planned PCLs](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl) and modify the dates below to accurately reflect the plan of action.
- [ ] If there is a Family & Friends day this month, add notes for pausing
  deployments before the day starts, and unpausing them before the next business day. See
  [the documentation](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#do-we-deploy-on-weekends-or-holidays).
- [ ] Update this issue with three planned dates for [recurring Staging rollback practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md). Consider spreading these across timezones to share the knowledge.
  - [ ] Set Due Date for this Issue to the first practice session
- [ ] Check for any [deprecations](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/package/libraries/deprecations.rb) and see if we are possibly affected in our [k8s](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/tree/master/releases/gitlab/values) or [chef](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/tree/master/roles) config.
  with `removal: '15.7'` that might affect our configuration -
  otherwise the next auto-deploy after a monthly release might fail with a deprecation
  failure

## First Staging Rollback Practice
### Date to be Completed:

- [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md)
- [ ] Set Due date for this issue to the date of the second practice session

## Second Staging Rollback Practice
### Date to be Completed:

- [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md)
- [ ] Set Due date for this issue to the date of the third practice session

## Third Staging Rollback Practice
### Date to be Completed:

- [ ] [Perform Staging Rollback Practice](https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/recurring-rollback-practice.md)
- [ ] Set Due date for this issue to the date of the release day

## Up until the 17th

- Ensure any deploys that do not make it to canary are investigated. [Disable canary if necessary].
- Push any successful deploy to canary into production after some time has passed (preferably 1h).
- Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
  1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
  1. If permission is granted, utilize the following command to initiate an overridden promotion:
    ```
    /chatops run deploy  gprd --ignore-production-checks 'deployment approved by on call SRE'
    ```
  1. This will post a comment into this issue and begin the deployment
  1. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

## 17th

If this date is on a weekend, do this work on the next working day

- [ ] [Find the latest `sha`](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/auto-deploy.md#auto-deploy-status) that made it into production successfully: `sha`
- [ ] Make sure to [execute the post-deploy migration pipeline] to ensure that all post-deploy migrations have been executed:
  `/chatops run post_deploy_migrations execute`.
- [ ] Notify Engineering Managers and developers that this is the `sha` that is guaranteed to be released on the 22nd:
    ```
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd.
    https://gitlab.com/gitlab-org/security/gitlab/commits/.
    You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 15.7`
    Please see the following documentation on what this means:
      * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
      * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
      * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
    ```

## 18th

If this date is on a weekend, do this work on the last Friday before the 18th.

- [ ] Log latest auto-deploy branch: **BRANCH_NAME**
- [ ] Ensure this build makes it through into production
- [ ] Make sure to [execute the post-deploy migration pipeline] to ensure that all post-deploy migrations have been executed:
  `/chatops run post_deploy_migrations execute`.
- [ ] Grab the `sha` from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate `sha` for the release:
    ```
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd.
    https://gitlab.com/gitlab-org/security/gitlab/commits/
    You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 15.7`
    Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
      * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
      * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
      * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
    ```

## 20th: two working days before the release

If this date is on a Sunday, do this work on the last Friday before the 20th.
If it falls on a Friday or Saturday, move it to Thursday.

- [ ] Determine what is the last auto deploy branch to have deployed to production and add it here: `BRANCH`
- [ ] If you plan to use the latest commit deployed to production to create the RC, make sure to
[execute the post-deploy migration pipeline] to ensure that all post-deploy migrations have been executed:
`/chatops run post_deploy_migrations execute`.
- [ ] Create a RC version to ensure that the final version builds correctly

   ```sh
   # In Slack:
   /chatops run release tag 15.7.0-rc42
   ```

This will use the latest commit deployed to production for the various
components that we release. If a different commit is necessary for a component,
such as GitLab, you should run the following instead:

```sh
/chatops run release tag 15.7.0-rc42 --gitlab-sha=XXX
```

This will then use `XXX` as the SHA to create the GitLab stable branches.

**NOTE:** this SHA is only used if the stable branch has yet to be created. If
it already exists, the branch is left as-is.

- [ ] Verify that the [CE stable branch] contains the right commits
  - There should be at least two commits: the last commit from the previous
    stable branch (usually a version update), and the sync commit created by the
    merge train.
  - The sync commit will have the message "Add latest changes from gitlab-org/gitlab@15-7-stable-ee"
- [ ] Verify that the pipelines are green

   ```sh
   # In Slack:
   /chatops run release status 15.7.0-rc42
   ```
- [ ] Notify Engineering Managers and developers that final candidate has been created:
    ```
    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd.
    https://gitlab.com/gitlab-org/security/gitlab/-/commits/15-7-stable-ee
    You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 15.7`
      * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
    ```

- [ ] Verify that the RC has been deployed to the [pre environment](https://pre.gitlab.com/help)

  - Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged.
    A notification will be sent to the `#announcements` channel in Slack when it starts.
  - If required to deploy manually, follow the steps in
    [pre-and-release-environments.md#manual-deployments](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/pre-and-release-environments.md#manual-deployments).

## 21st: one day before the release

If this date is on a weekend, do this work on the Friday before that weekend.

- [ ] Confirm that final RC version has passed automated tests
  - [ ] Ensure tests are green on [CE stable branch]
  - [ ] Ensure tests are green on [EE stable branch]
  - [ ] Ensure tests are green on [Omnibus]
  - [ ] Ensure default and stable branches are synced: `/chatops run mirror status`
- [ ] Tag `15.7.0`:
    ```sh
    # In Slack:
    /chatops run release tag 15.7.0
    ```
  - [ ] Check progress of [EE packages build](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+ee.0) and [CE packages build](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+ce.0)
- [ ] Validate `15.7.0` has been deployed to the [release environment](https://release.gitlab.net/help)

Instructions for manual deploy

        ```sh
        # In Slack:
        /chatops run deploy 15.7.0-ee.0 release
        ```



- [ ] Validate `15.7.0` has been passed automated QA by ensuring the `release-gitlab-qa-smoke` job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

## 22nd: release day

Final release is tagged, so any changes will have to initiate a patch release. [Reminder: We have a soft PCL today](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl), coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

- [ ] At 13:00 UTC, post an update about the package building status in `#f_upcoming_release`
   ```
   :mega: Packages for 15.7.0 are built and will be published at 13:10UTC
   ```
- At 13:10 UTC:
  - :warning: **Make sure that neither packages nor the blog post get published earlier than 13:10UTC
    without approval by the
    [messaging lead](https://about.gitlab.com/handbook/marketing/blog/release-posts/managers/)
    of the release post. Mind that you don't need their approval if you're on time** :warning:
  - [ ] Publish the packages via ChatOps:
    ```
    # In Slack:
    /chatops run publish 15.7.0
    ```
  - If anything goes wrong and the release is delayed, ping the
  [release post manager](https://about.gitlab.com/handbook/marketing/blog/release-posts/managers/)
  on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
- At 14:10 UTC:
  - [ ] Verify the `check-packages` job completes successfully on the [EE Pipeline](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+ee.0)
  - [ ] Verify the `check-packages` job completes successfully on the [CE Pipeline](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+ce.0)
  - [ ] Verify that Docker images appear on `hub.docker.com`: [EE](https://hub.docker.com/r/gitlab/gitlab-ee/tags) / [CE](https://hub.docker.com/r/gitlab/gitlab-ce/tags)
  - [ ] Post an update about the status in `#f_upcoming_release`
  ```
  :mega: 15.7.0 is published and publicly available
  ```
  - [ ] Once all packages are available publicly and GitLab.com is up and running on the release version,
  ping the [release post manager](https://about.gitlab.com/handbook/marketing/blog/release-posts/managers/)
  on Slack ([#release-post channel]) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC
  - [ ] Create the `15.7.0` version on [version.gitlab.com](https://version.gitlab.com/versions/new?version=15.7.0)

## Release Certification

The [release certification process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/release-certification.html) may apply to this release. cc @gitlab-com/gl-security/federal-application-security

[CE stable branch]: https://gitlab.com/gitlab-org/gitlab-foss/commits/15-7-stable
[EE stable branch]: https://gitlab.com/gitlab-org/gitlab/commits/15-7-stable-ee
[Omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/commits/15-7-stable
[Deploy]: https://gitlab.com/gitlab-org/takeoff#deploying-gitlab
[staging.gitlab.com]: https://staging.gitlab.com/
[getting help]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/monthly.md#getting-help
[#release-post channel]: https://gitlab.slack.com/messages/C3TRESYPJ
[Disable canary if necessary]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/canary.md#how-to-stop-all-production-traffic-to-canary
[Adjust the auto-deploy branch]: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/how-to-set-auto-deploy-branch-schedule.md
[execute the post-deploy migration pipeline]: https://gitlab.com/gitlab-org/release/docs/-/tree/master/general/post_deploy_migration#how-to-execute-post-deploy-migrations

2022-12-20 11:23:25.923075 I [dry-run] ReleaseTools::PickIntoLabel -- Creating monthly Pick label -- {:label=>"Pick into 15.7"}

Release candidate (no change) release:prepare[15.7.0-rc4]

Uses the release_candidate.md.erb template as expected. This task failed, but only because this rc version does not exist, so attempts to fetch related branches failed, which is expected.


~/workspace/gitlab-org/release-tools (delivery2731-versionless-patch-prepare ✗) TEST='test' RELEASE_BOT_PRODUCTION_TOKEN=$TOKEN RELEASE_BOT_OPS_TOKEN=$OPS_TOKEN be rake release:prepare\[15.7.0-rc4\]

# Release 15.7.0-rc4



## Packaging

- [ ] Tag `15.7.0-rc4`:

   ```sh
   # In Slack:
   /chatops run release tag 15.7.0-rc4
   ```
- [ ] Check progress of [EE packages build](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+rc4.ee.0) and [CE packages build](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+rc4.ce.0)
    - This might take a while (around 80 min).
    - We only need the EE packages to finish to continue with next steps.

## Deploy

### staging.gitlab.com

- Staging deploys via the [deployer pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/deployer) happen automatically as soon as the
  [EE packages build](https://dev.gitlab.org/gitlab/omnibus-gitlab/commits/15.7.0+rc4.ee.0)
  reaches the `gitlab_com:upload_deploy` stage of the pipeline.

### canary stage gitlab.com

- Canary deploys via the [deployer pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/deployer) happen automatically as soon as the
  QA jobs have completed successfully
- [ ] Confirm that there are no errors on canary
  - [canary errors on sentry.gitlab.net](https://sentry.gitlab.net/gitlab/gitlabcom/?query=server_name%3A%22web-cny-01-sv-gprd%22)
  - [canary dashboard](https://dashboards.gitlab.net/d/llfd4b2ik/canary)

**If there are issues on canary you should immediately stop sending traffic to it by issuing the following chatops command**:

```
/chatops run canary --drain --production

```

- This package shall not reach the rest of the production environment as we want to keep production on the auto-devops process

## Release

- [ ] Publish the packages via ChatOps:
  ```
  # In Slack:
  /chatops run publish 15.7.0-rc4
  ```
- [ ] Verify that packages appear on [`packages.gitlab.com`](https://packages.gitlab.com/gitlab/unstable)
- [ ] Verify that Docker images appear on `hub.docker.com`: [EE](https://hub.docker.com/r/gitlab/gitlab-ee/tags) / [CE](https://hub.docker.com/r/gitlab/gitlab-ce/tags)

[Preparation MRs]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/picking-into-merge-requests.md
[remaining merge requests]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=Pick%20into%2015.7
[Deploy]: https://gitlab.com/gitlab-org/takeoff#deploying-gitlab
[announce the deploy]: https://gitlab.com/gitlab-org/takeoff/blob/master/doc/announce-a-deployment.md

# Draft: Prepare 15.7.0-rc4-ee release

2022-12-20 11:29:57.606387 D [dry-run] ReleaseTools::GitlabClient -- [HTTParty] [2022-12-20 11:29:57 -0700] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Frelease%2Ftasks/issues" -
2022-12-20 11:29:57.607080 F [dry-run] Rake::Task -- Task failed -- Exception: NoMethodError: undefined method `web_url' for nil:NilClass

☑ Author Check-list

[-] Has documentation been updated?

Edited Dec 23, 2022 by Steve Abrams

Release preparation extended maintenance policy

🌲 What does this MR do and why?

📽 Screenshots

☑ Author Check-list

Merge request reports