Skip to content

Kubernetes namespace management for review environments with the agent for Kubernetes

Background

GitLab-managed clusters are currently the easiest way for platform engineers to facilitate arbitrary deployments for the developers/teams they support. While this approach offers a lot of flexibility for developers deploying their projects, it can lead to pain points for platform engineers particularly on multi-tenant clusters:

  • requires giving GitLab extensive permissions on the cluster (cluster-admin rolebinding)
  • no configurability of the conventions used to create namespaces per project/environment scope
  • no way to limit the permissions granted to the service account used in the deployments (admin rolebinding)
  • no easy way to manage cluster integrations across multiple groups/projects, requires duplicating configuration
  • no way to hide cluster configuration from group/project maintainers

As far as I can tell, roadmap for agentk has primarily focused on the git-ops use case, a very necessary starting point, but has not attempted to facilitate cluster integrations as they already exist today (#220911 (closed)).

Use Case

As a platform engineer, I would like to leverage agentk to provide a consistent platform for developers to deploy on top of. In addition to using git-ops for provisioning baseline resources consistently across the clusters I manage, I need to enable developers to deploy their projects on these multi-tenant clusters in a predictable and compliant way.

Issues with the existing agent-based CI/CD workflow

  • How to configure KUBE_CONTEXT when 2-3 projects are created every week and the deployments happen to different environments, clusters based on the job and branch?
    • They can try using CI variables in the agent-name part of the KUBE_CONTEXT?
  • How can they use the agent when every new project needs its own Service Account? How can they make Service Account setup automatic in each environment for every new project?

Adding necessary manual work to the processes is not an option.

Requirements

When an agent tunnel is shared at the group level
Given that a deployment happens from a project
Then create and use a namespaces following the pattern given in the agent configuration
When an agent tunnel is shared at the project level
Given that a deployment happens from the project
Then restrict the deployment to the namespace given in the agent configuration

Service account requirements are already handled with the impersonation features.

Proposal 1

Introduce a namespace_pattern key that assures the given namespace exists, and sets it for the CI tunnel impersonation to allow for RBAC targeting. The namespace_pattern can use the predefined CI variables for dynamic namespace names.

ci_access:
  groups:
  - id: path/to/group/subgroup
    namespace_pattern: 'team-{CI_PROJECT_NAME}"

Proposal 3

Investigate approaches to scriptable agent beha... (gitlab-org/cluster-integration/gitlab-agent#103)

Proposal 4

Use statically created namespaces and service accounts, and use them given their annotations as described in #280563 (comment 483750616)

Proposal 5

Introduce a tool like Kyverno or an admission controller that can generate resources in the cluster based on a specific conditions. A Kyverno example can be seen at https://kyverno.io/policies/flux/generate-flux-multi-tenant-resources/generate-flux-multi-tenant-resources/

Proposal 6

Ship an Application controller and a CRD that can be configured to set up an environment in a templated way. We can leverage existing tools, like Crossplane to provide the Application CRD or come up with our own wrapper or implementation.

Open Questions

  • should we provide special configuration blocks for how agentk creates resources for deployment targets?
  • do we instead just require things like a namespace and service account be defined in a manifest project statically and do not have agentk create anything at deploy time?
    • perhaps we could look at annotations on resources in the manifest project to make agentk aware of which namespace and service account to use for a given project?
    • edit: see #280563 (comment 483750616) for an example
  • depending on the outcome of &4516 (closed), how do we manage environment scope(s) for the target groups/projects?
  • how could a target group/project use the git-ops model if configured this way?
  • does this conflict with design goals regarding multiple agents per cluster and/or one agent across multiple clusters?

Pricing

/cc @nagyv-gitlab @tkuah @ash2k

Edited by Viktor Nagy (GitLab)