Skip to content

Update documentation for package counts

John Skarbek requested to merge jts/update-package-counts-again into master

What does this MR do?

  • Packages created are always different between differing versions
  • Maintaining a list doesn't make sense
  • Instead, I provide an example of how to determine how many packages SHOULD exist
  • This provides us with better confidence when needing to validate that package publishing worked as desired

Patch ERB test

Click to Expand

Release 42.42.42

Preparation

  • Preparation MR's should already be created
  • Ensure 42-42-stable-ee GitLab branch is green.
  • Ensure any backports targeting 42.42.42 are merged to their stable counter part
  • Perform automated merging into the preparation branches: 
    # In Slack
/chatops run release merge 42.42.42
  • Check for any MR's that might have been created that are targeting our preparation branch
  • Merge the preparation merge requests
    • gitlab-org/gitlab
    • gitlab-org/omnibus-gitlab
    • gitlab-org/build/CNG
  • Check the following list of critical issues/MRs which are to be included in 42.42.42.
    • REFERENCE_TO_MR_TO_PICK
  • Ensure builds are green on Omnibus

Backport request

If this patch release is part of a backport request, we will not be able to deploy the package to our release instance. Now is the time to kick off QA. Refer to the backport QA testing documentation to complete this task:

  • QA is complete, and sign-off by the current QA on-call is noted as a comment on this issue.

Packaging

  • Check if mirroring synced stable branches to dev. If the output is ✅ for every repo, we can proceed to tag. Note. If GitLab Canonical to Security mirroring has diverged due to security merges this mirror is expected to show as a broken and can be safely ignored.

    # In Slack
/chatops run mirror status

  • Tag 42.42.42:

    # In Slack:
/chatops run release tag 42.42.42

  • While waiting for packages to build, now is a good time to prepare the blog post. Look at previous MRs for examples. => BLOG_POST_MR

  • Check progress of EE packages build and CE packages build.

    • This might take a while (around 80 min).
    • We only need the EE packages to finish to continue with next steps.

Deploy

For patch releases, the only available environment for deploys is release.gitlab.net. All GitLab Inc. team members can login to that installation using their email address (through google oauth).

Deployment to release.gitlab.net is not required if the patch release is not for the latest completed monthly release version. This is because release.gitlab.net will be on the latest monthly release, and deploying a patch of an older version will require rolling back release.gitlab.net.

release.gitlab.net

Deployments to release.gitlab.net are performed automatically.

Instructions to manually deploy if required.

If you need to manually run a deployment, you can do so as follows:

# In Slack:
/chatops run deploy 42.42.42-ee.0 --release

Release

  • Publish the packages via ChatOps: 
    # In Slack:
/chatops run publish 42.42.42
  • Verify that EE packages appear on packages.gitlab.com: EE
  • Verify that CE packages appear on packages.gitlab.com: CE
Package Counts

Inside of the omnibus-gitlab pipelines, the Package-and-image-release stage contain many jobs. For all jobs that are part of an Operating System Package, count each of them one time, with the exception of CentOS 7, count this one 3 times.

For example, for 14.9.X-ee:

  • AmazonLinux-2-arm64-release
  • AmazonLinux-2-release
  • CentOS-7-release
  • CentOS-8-arm64-release
  • CentOS-8-release
  • Debian-9-release
  • Debian-10-arm-release
  • Debian-10-release
  • Debian-11-arm-release
  • Debian-11-release
  • OpenSUSE-15.3-arm64-release
  • OpenSUSE-15.3-release
  • SLES-12.2-release
  • SLES-12.5-release
  • SLES-15.2-release
  • Ubuntu-18.04-release
  • Ubuntu-20.04-arm-release
  • Ubuntu-20.04-release

The above would produce 20 total packages.

Points of importance:

  • The amount and type of these jobs changes over time, therefore the number of packages created may change with each release
  • Reach out to the Distribution team for any questions related to needing to confirm packages
  • CE normally has fewer packages
  • Verify that Docker images appear on hub.docker.com: EE / CE
  • Deploy the blog post
  • Create the 42.42.42 version on version.gitlab.com

References

gitlab.com

dev.gitlab.org

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security

/milestone %"42.42" /due in 7 days

Monthly ERB test

Click to expand

Release 42.42

First steps

  • Change #f_upcoming_release topic with /topic 42.42.0: <link_to_this_issue>
  • [Adjust the auto-deploy branch] schedule based on the Release Manager's working hours.
  • Consider planned PCLs and modify the dates below to accurately reflect the plan of action.
  • If there is a Family & Friends day this month, add notes for pausing deployments before the day starts, and unpausing them before the next business day. See the documentation.
  • Update this issue with three planned dates for recurring Staging rollback practice. Consider spreading these across timezones to share the knowledge.
    • Set Due Date for this Issue to the first practice session
  • Check for any deprecations and see if we are possibly affected in our k8s or chef config. with removal: '42.42' that might affect our configuration - otherwise the next auto-deploy after a monthly release might fail with a deprecation failure

First Staging Rollback Practice

Date to be Completed:

Second Staging Rollback Practice

Date to be Completed:

Third Staging Rollback Practice

Date to be Completed:

Up until the 17th

  • Ensure any deploys that do not make it to canary are investigated. [Disable canary if necessary].
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, utilize the following command to initiate an overridden promotion:
    /chatops run deploy <VERSION> --production --ignore-production-checks 'deployment approved by on call SRE'
    1. This will post a comment into this issue and begin the deployment
    2. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

17th

If this date is on a weekend, do this work on the next working day

  • Find the latest sha that made it into production successfully: sha
  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 22nd: 
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 42.42`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

18th

If this date is on a weekend, do this work on the last Friday before the 18th.

  • Log latest auto-deploy branch: BRANCH_NAME
  • Ensure this build makes it through into production
  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release: 
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA>
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 42.42`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

20th: two working days before the release

If this date is on a Sunday, do this work on the last Friday before the 20th. If it falls on a Friday or Saturday, move it to Thursday.

  • Determine what is the last auto deploy branch to have deployed to production and add it here: BRANCH

  • Create a RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 42.42.0-rc42

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 42.42.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

  • Verify that the [CE stable branch] contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@42-42-stable-ee"

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/42-42-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 42.42`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

21st: one day before the release

If this date is on a weekend, do this work on the Friday before that weekend.

  • Confirm that final RC version has passed automated tests
    • Ensure tests are green on [CE stable branch]
    • Ensure tests are green on [EE stable branch]
    • Ensure tests are green on Omnibus
    • Ensure default and stable branches are synced: /chatops run mirror status
  • Tag 42.42.0: 
    # In Slack:
/chatops run release tag 42.42.0
  • Validate 42.42.0 has been deployed to the release environment
Instructions for manual deploy 
    ```sh
    # In Slack:
    /chatops run deploy 42.42.0-ee.0 --release
    ```
  • Validate 42.42.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

22nd: release day

Final release is tagged, so any changes will have to initiate a patch release.

  • At 13:00 UTC, post an update about the package building status in #f_upcoming_release

    :mega: Packages for 42.42.0 are built and will be published at 13:30UTC

  • At 13:30 UTC:

    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:30UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • Publish the packages via ChatOps: 
      # In Slack:
/chatops run publish 42.42.0
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too

  • At 14:10 UTC:

    • Verify that EE packages appear on packages.gitlab.com: EE
    • Verify that CE packages appear on packages.gitlab.com: CE
    Package Counts

Inside of the omnibus-gitlab pipelines, the Package-and-image-release stage contain many jobs. For all jobs that are part of an Operating System Package, count each of them one time, with the exception of CentOS 7, count this one 3 times.

For example, for 14.9.X-ee:

  • AmazonLinux-2-arm64-release
  • AmazonLinux-2-release
  • CentOS-7-release
  • CentOS-8-arm64-release
  • CentOS-8-release
  • Debian-9-release
  • Debian-10-arm-release
  • Debian-10-release
  • Debian-11-arm-release
  • Debian-11-release
  • OpenSUSE-15.3-arm64-release
  • OpenSUSE-15.3-release
  • SLES-12.2-release
  • SLES-12.5-release
  • SLES-15.2-release
  • Ubuntu-18.04-release
  • Ubuntu-20.04-arm-release
  • Ubuntu-20.04-release

The above would produce 20 total packages.

Points of importance:

  • The amount and type of these jobs changes over time, therefore the number of packages created may change with each release
  • Reach out to the Distribution team for any questions related to needing to confirm packages
  • CE normally has fewer packages

Security patch template

Click to expand

Security patch release: 14.9.3, 14.8.6, 14.7.8

First steps

Early-merge phase

Up until the 27th, or one day before the Security Release due date

  • Merge the merge requests targeting default branches

    # In Slack
/chatops run release merge --security --default-branch

  • Verify if a Gitaly security fix is included in the upcoming security release, if it is, follow the How to deal with Gitaly security fixes guide.

On the 27th (one day before due date)

If this date is on a weekend, do this work on the next working day.

  • Determine the security release manager from the schedule. Look for the security release manager of the latest released monthly version.

  • Post the following message to #sec-appsec in Slack: <security-release-manager> We are starting the [security release](<link to this issue>), aiming for release tomorrow. Please create a blog post MR on gitlab-org/security/www-gitlab-com.

  • Once the blog post MR has been created by the security release manager, add a link to it here: https://gitlab.com/gitlab-org/security/www-gitlab-com/-/merge_requests/

  • Merge security merge requests targeting default branches

    # In Slack:
/chatops run release merge --security --default-branch

  • Check that all MRs merged into the default branch have been deployed to production:

    # In Slack:
/chatops run auto_deploy security_status

    NOTE: This only checks gitlab-org/security/gitlab. If other projects have security MRs you should verify those manually.

  • Merge backports and any other merge request pending:

    # In Slack:
/chatops run release merge --security

  • Ensure all Merge Requets associated with Implementation Issues with label reduced backports are merged

  • Unlink any security issues that are not ready from the security release tracking issue (in gitlab-org/gitlab), and post a comment listing the issues that have been removed, why they have been removed (the reason returned by the run release merge command), and ping the authors.

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • If there are no Omnibus security issues associated, consider tagging today already.

    • Omnibus default branch MRs are only merged together with backport MRs and need to be deployed to gprd first before tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • Tag the 14.9.3 security release, and wait for the pipeline to finish: /chatops run release tag --security 14.9.3

  • Tag the 14.8.6 security release, and wait for the pipeline to finish: /chatops run release tag --security 14.8.6

  • Tag the 14.7.8 security release, and wait for the pipeline to finish: /chatops run release tag --security 14.7.8

Waiting between pipelines is necessary as they may othewise fail to concurrently push changes to the same project/branch.

Deploy

Release

  • Publish packages via ChatOps: 
    /chatops run publish --security
  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/
  • Verify that EE packages appear on packages.gitlab.com: EE (should contain 15 packages)
  • Verify that CE packages appear on packages.gitlab.com: CE (should contain 13 packages)
Package Counts

Inside of the omnibus-gitlab pipelines, the Package-and-image-release stage contain many jobs. For all jobs that are part of an Operating System Package, count each of them one time, with the exception of CentOS 7, count this one 3 times.

For example, for 14.9.X-ee:

  • AmazonLinux-2-arm64-release
  • AmazonLinux-2-release
  • CentOS-7-release
  • CentOS-8-arm64-release
  • CentOS-8-release
  • Debian-9-release
  • Debian-10-arm-release
  • Debian-10-release
  • Debian-11-arm-release
  • Debian-11-release
  • OpenSUSE-15.3-arm64-release
  • OpenSUSE-15.3-release
  • SLES-12.2-release
  • SLES-12.5-release
  • SLES-15.2-release
  • Ubuntu-18.04-release
  • Ubuntu-20.04-arm-release
  • Ubuntu-20.04-release

The above would produce 20 total packages.

Points of importance:

  • The amount and type of these jobs changes over time, therefore the number of packages created may change with each release
  • Reach out to the Distribution team for any questions related to needing to confirm packages
  • CE normally has fewer packages
  • Verify that Docker images appear on hub.docker.com: EE / CE
  • Deploy the blog post
  • Create the versions:

Final steps

  • Sync default branches for GitLab, GitLab Foss, Omnibus GitLab and Gitaly, via ChatOps:

    # In Slack
/chatops run release sync_remotes --security

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

  • Close the security implementation issues

    # In Slack
/chatops run release close_issues --security

  • Notify engineers the security release is out (blog post link needs to be replaced with the actual link):

    /chatops run notify ":mega: GitLab Security Release: 14.9.3, 14.8.6, 14.7.8 has just been released: <blog post link>! Share this release blog post with your network to ensure broader visibility across our community."

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules

  • In case it was disabled, enable the Gitaly update task.

  • Close the old security release tracking issue and create a new one:

    # In Slack
/chatops run release tracking_issue --security

  • Link the new security release tracking issue on the topic of the #releases channel, next to Next Security Release.

Author Check-list

  • Has documentation been updated?
Edited by Reuben Pereira

Merge request reports