Merge injected KUBECONFIGs for clusters and agents
What does this MR do and why?
This makes it easier to migrate one environment at a time from a certificate-based connection to an agent-based connection by
- switching just that environment's context
- and explicitly setting the default namespace to match the environment's expected namespace
See #335089 (closed)
Screenshots or screen recordings
With the following .gitlab-ci.yml
# .gitlab-ci.yml
test:
environment: test
image:
name: bitnami/kubectl
entrypoint: [""]
script:
- kubectl config get-contexts
- kubectl config current-context || echo "No current context"
- kubectl get pods || echo "Could not get pods"
- kubectl config use-context cluster-group/agent-config:agent || echo "Context does not exist"
- kubectl config current-context
- kubectl --insecure-skip-tls-verify=true get namespaces || echo "Unable to get namespaces"
- cat $KUBECONFIG
cluster enabled | cluster disabled | |
---|---|---|
agent enabled | ||
agent disabled |
How to set up and validate locally
For a full E2E test like the one above:
Prerequisites
One-time setup steps required to get E2E testing with GitLab CI and a local Kubernetes cluster working:
-
Install a docker runtime. If you are on macOS, I recommend colima (
brew install colima
). -
Install
kubectl
(brew install kubectl
) -
Install kind (
brew install kind
). We will use this to create the test cluster -
Configure a loopback interface for your GDK. Then add
kubernetes
as an additional binding to the loopback IP in/etc/hosts
. For example, if172.16.123.1
is the IP, then append this to/etc/hosts
:172.16.123.1 kubernetes
-
Install https://gitlab.com/hfyngvason/kubectl-gitlab/. This makes it easier to add the cluster as a certificate-based cluster.
-
In the terminal session you are going to be using, run
export GITLAB_URL=<YOUR GDK URL> export GITLAB_TOKEN=<A PERSONAL API TOKEN FOR YOUR GDK> # the leading space is intentional so it doesn't get saved in your shell history
-
Verify that
kubectl-gitlab
is working: Runkubectl gitlab -i ls
to see your instance-level clusters. It should return an empty array (unless you have a cluster configured). -
Enable HTTPS in your GDK. This is required for the CI/CD tunnel:
gdk config set https.enabled true gdk reconfigure gdk restart
-
Configure a docker runner in your GDK. Note that this requires gitlab-development-kit!2462 (merged) to function smoothly with HTTPS (currently being merged).
Testing with a cluster
Goal: Reproduce the upper-left quadrant screenshot above with cluster enabled and agent enabled
- Create a cluster with
kind
. Save the following askind.yaml
# kind.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: apiServerAddress: 0.0.0.0
kind create cluster --config kind.yaml # change the cluster hostname to kubernetes. This way, we can address the cluster from within a docker container sed -i -s 's/0\.0\.0\.0/kubernetes/g' "$HOME/.kube/config"
kubectl cluster-info
- Add the cluster as an instance-level cluster to your local GDK. Important:
GITLAB_URL
andGITLAB_TOKEN
must be set in your environment as described above.kubectl gitlab add -i
- Install the GitLab Agent to your cluster:
- Create a blank project, add a blank file
.gitlab/agents/agent/config.yaml
- Navigate to
Infrastructure > Kubernetes
, clickActions > Connect with an agent
, selectagent
from the dropdown and copy-paste thedocker run ...
command into your terminal.
- Create a blank project, add a blank file
- We now need to amend the deployment with self-signed certificate for GitLab:
- Create a configmap for your self-signed certificate:
kubectl -n gitlab-kubernetes-agent create configmap gdk-ca --from-file "localhost.crt"
- Edit the deployment:
kubectl -n gitlab-kubernetes-agent edit deployment gitlab-agent
containers[0].args
(just below a line similar to- wss://gdk.test:3000/-/kubernetes-agent
), append- --ca-cert-file=/gdk-ca/localhost.crt
volumeMounts:
append:- mountPath: /gdk-ca name: gdk-ca
volumes:
append:- configMap: defaultMode: 420 name: gdk-ca name: gdk-ca
- Create a configmap for your self-signed certificate:
- Refresh the agents page. The agent should now show as
Connected
underConnection status
in the table. - Add the following
.gitlab-ci.yml
to the project:test: environment: test image: name: bitnami/kubectl entrypoint: [""] script: - kubectl config get-contexts - kubectl config current-context || echo "No current context" - kubectl get pods || echo "Could not get pods" - kubectl config use-context cluster-group/agent-config:agent || echo "Context does not exist" - kubectl config current-context - kubectl --insecure-skip-tls-verify=true get namespaces || echo "Unable to get namespaces" - cat $KUBECONFIG
- A pipeline should run, and the output should look similar to the screenshot above.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.