Skip to content

Investigate deploying review apps using Helm 3

To investigate what it takes to deploy review app using Helm 3.

From team meeting discussion:

Distribution issue:

Remy: We need to update CI build image (registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-charts-build-base) - we are currently using the same image between review-apps job and charts gitlab project Split the current image to a helm2 image (the current one) and a new helm3 image Kyle: keep in mind of helm 2 specific steps in the review deploy job

Todo:

  • New CI image with Helm 3.1.2 and kubectl 1.18.0: gitlab-build-images!282 (merged)
  • Verify current deployment script with Helm 3 CLI
  • Verify clean up script with Helm 3 CLI

Questions

  • Rollout plan for helm 3
  • can helm 3 releases coexist with helm 2 releases in the same k8s namespace?
  • can 2 external-dns (1 by helm 2 and 1 by helm 3) coexist in the same k8s namespace?
  • how would clean up be done? would helm 3 clean up releases created by helm 2?
  • do we need to migrate helm 2 release to helm 3?

Findings

  1. Concurrent deployments from helm2 and helm3 into the same namespace is possible. I had an issue with repeated certmanager installation, but since review app is using its own TLS certificate, this wouldn't be an issue
  2. Changes needed to related scripts:
    • review-deploy:
      • --timeout option now requires unit: change 900 to 900s
      • remove install_tiller()
      • install_external_dns:
    • review-stop-failed-deployment:
      • replace --tiller-namespace with --namespace in previous_deploy_failed()
    • review-stop
      • change delete --purge to uninstall
      • replace --tiller-namespace with namespace
    • review-cleanup:
      • Quality::HelmClient:
        • remove tiller_namespace from
        • change delete --purge to uninstall
        • remove --tiller-namespace in #raw_releases
        • change parsing of helm list json output into Release struct
          • helm2: {"Next":"","Releases":[{"Name":"glh2","Revision":1,"Updated":"Wed Apr 1 13:29:09 2020","Status":"DEPLOYED","Chart":"gitlab-3.2.1","AppVersion":"12.9.1","Namespace":"gitlab"}]}
          • helm3: [{"name":"glh3","namespace":"gitlab","revision":"1","updated":"2020-04-01 13:33:31.716462 +0800 +08","status":"deployed","chart":"gitlab-3.2.1","app_version":"12.9.1"}]
  3. Changes needed to configuration:
  4. helm2 and helm3 keeps track of release history independently, so only helm2 can remove / upgrade helm2 releases and vice versa, unless data is migrated.

Proposed roll out plan

review-apps-ee

  1. #213197 (closed) Create "helm upgrade" MR to change scripts to helm 3. This MR would:
    1. Change review-deploy, review-stop, review-stop-failed-deployment to use helm 3
    2. Keep review-cleanup to use helm 2 for cleaning up helm 2 review apps
    3. Delete dns-gitlab-review-app manually
    4. Install a new dns-gitlab-review-app using helm 3 into the same release name dns-gitlab-review-app as part of review_deploy
  2. #213198 (closed) Create "helm3 clean up" MR to enable clean up of helm 3 tracked review apps
    1. Create new job review-cleanup-helm3 to clean up helm 3 review apps
    2. Change Quality::HelmClient to be able to support helm 2 and helm 3 CLI #212529 (comment 315350954))
  3. Merge "helm upgrade" MR
    1. Delete master review app helm 2 installation (see #212529 (comment 315355338))
    2. Next scheduled pipeline will install master review app using helm 3
    3. Scheduled clean up will clean up helm 2 review apps
  4. Merge "helm3 clean up" MR
    1. At this point, helm 2 and helm 3 releases will be cleaned up by respective jobs
  5. Existing open MRs will continue to use helm 2 to install review app until they are rebased with master
    • Upon rebase, helm 3 would attempt to install into the same release name that was previously used by helm 2. If there is an existing helm 2 deployment, this could fail, and requires deleting the helm 2 release first, as noted in #212529 (comment 316837035)
  6. New open MRs will use helm 3 to install review app
  7. Monitor review apps in helm 2 release list
  8. #213199 (closed) Deprecate helm 2 when there is no more releases - create new "deprecate helm 2" MR to:
    1. Rename review-cleanup-helm3 to review-cleanup and remove the old review-cleanup job
    2. Clean up Quality::HelmClient
    3. clean up tiller

review-apps-ce

  1. It shouldn't take additional work, as there are no MRs to this repo 1.After the change from "helm upgrade" MR is pulled into gitlab-foss, the pipeline would start using helm 3.
    1. Delete dns-gitlab-review-app manually
    2. Install a new dns-gitlab-review-app using helm 3 into the same release name dns-gitlab-review-app (see #212529 (comment 315350954)) as part of review_deploy
  2. After the change from "deprecate helm 2" MR is pulled into gitlab-foss
    1. clean up tiller

/cc @gl-quality/eng-prod

Edited by Albert Salim