Skip to content

Auto DevOps using Agent K with multiple clusters

Summary

This tickets lies a bit between a refactoring, feature request and broken feature report.

As described in this note on ticket #297004 (closed), moving from the certificate based cluster management with multiple environments (staging, testing, production each on its own cluster), to using Agent K in a similar fashion currently does not work. From what I found, this has to do with the Auto-DevOps.gitlab-ci.yml template handling only a single KUBE_CONTEXT variable for all the jobs. Without knowing the internals of GitLab itself, I would venture that this variable is likely set based on the environment defined for each job during a pipeline run with multiple clusters connected.

There are discussions to add support for environments to the GitLab Agent but it does not seem to match what the certificate based management provides in terms of applying resources to different specific clusters based on environment configured per job.

One solution I have found that does not require fundamental code changes is to create a custom template that includes Auto-DevOps.gitlab-ci.yml and does the KUBE_CONTEXT, as well as several other key variables, handling.

Improvements

The suggestion I am thinking about is to either:

  1. Refactor the Auto-DevOps.gitlab-ci.yml in such a way that it can be used with both certificate based and Agent K based clusters
  2. Add a new template to use with Agent K to achieve some level of feature parity

The goal being to give GitLab users an option to migrate from the deprecated system to the new one with less friction.

If solution two would be implemented, it would allow to keep both templates close together making their maintenance easier.

One thing that is currently not clear with having a single template, as suggested by option one, is how to select the cluster management project when using Agent K and how to completely ignore that part when using the certificate based system.

Depending on the level of complexity, suggestion number 2 will likely be simpler. There's an example of it below.

Risks

The main risk I can see is some duplication to ensure all stages are properly handled by both templates.

Involved components

lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml

Intended side effects

If proposition number 1 is selected, it should allow for minimal breakage of users workflow when moving to Agent K.

If proposition number 2 is selected, it will require users that are using the "default to Auto DevOps" pipeline option to create a .gitlab-ci.yml file to include the new Agent K compatible template.

Missing test coverage

From the top of my head, I would say that the tests currently done for the certificate based system would need to be at least partly duplicated to ensure that the Agent K variant works the same.

Code example

This is a re-post of the content of the note cited above to have everything in one place. This is a partial implementation of proposition number 2 that takes into account a separate cluster management project. This implementation tries to be as flexible as possible with sane defaults. It also reuses naming convention from the original Auto DevOps pipeline to be used, as much as possible, as a drop-in replacement.

includes:
  - template: Auto-DevOps.gitlab-ci.yml

variables:
  AGENTS_MGMT_PROJECT: $CI_PROJECT_NAMESPACE/$CI_PROJECT_NAMESPACE-cluster-management
  KUBE_CONTEXT_TESTING: $AGENTS_MGMT_PROJECT:agentk-testing
  KUBE_CONTEXT_STAGING: $AGENTS_MGMT_PROJECT:agentk-staging
  KUBE_CONTEXT_PRODUCTION: $AGENTS_MGMT_PROJECT:agentk-production

  KUBE_INGRESS_CLUSTERS_DOMAIN: clusters.example.com
  KUBE_INGRESS_BASE_DOMAIN_TESTING: testing.$KUBE_INGRESS_CLUSTERS_DOMAIN
  KUBE_INGRESS_BASE_DOMAIN_STAGING: staging.$KUBE_INGRESS_CLUSTERS_DOMAIN
  KUBE_INGRESS_BASE_DOMAIN_PRODUCTION: production.$KUBE_INGRESS_CLUSTERS_DOMAIN

  KUBE_NAMESPACE_SLUG: $CI_PROJECT_PATH_SLUG-$CI_PROJECT_ID
  KUBE_NAMESPACE_TESTING: $KUBE_NAMESPACE_SLUG-testing
  KUBE_NAMESPACE_STAGING: $KUBE_NAMESPACE_SLUG-staging
  KUBE_NAMESPACE_PRODUCTION: $KUBE_NAMESPACE_SLUG-production

review:
  variables:
    KUBE_INGRESS_BASE_DOMAIN: $KUBE_INGRESS_BASE_DOMAIN_TESTING
    KUBE_CONTEXT: $KUBE_CONTEXT_TESTING
    KUBE_NAMESPACE: $KUBE_NAMESPACE_TESTING

stop_review:
  variables:
    KUBE_CONTEXT: $KUBE_CONTEXT_TESTING

staging:
  variables:
    KUBE_INGRESS_BASE_DOMAIN: $KUBE_INGRESS_BASE_DOMAIN_TESTING
    KUBE_CONTEXT: $KUBE_CONTEXT_STAGING
    KUBE_NAMESPACE: $KUBE_NAMESPACE_STAGING

# etc.

This example is based on a template that has been tested and is working with Agent K.

I'd be happy to provide a patch if interested.