Add configuration for the KAS CI tunnel feature (!5686) · Merge requests · GitLab.org / omnibus-gitlab

Hordur Freyr Yngvason requested to merge add-kas-k8s-proxy-configuration into master Oct 20, 2021

What does this MR do?

KAS has a new endpoint that can proxy calls from GitLab CI to the Kubernetes API server, but it runs on a different port and needs additional configuration to work.

Related issues

gitlab#324272 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

Merge Request Title, and Description are up to date, accurate, and descriptive
MR targeting the appropriate branch
MR has a green pipeline on GitLab.com
Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

Test plan indicating conditions for success has been posted and passes
Documentation created/updated
Tests added
Integration tests added to GitLab QA
[-] Equivalent MR/issue for the GitLab Chart opened
- This functionality has been in the chart for a while.

Test plan

We are going to set up a self-signed Omnibus accessible from the local machine under the host https://gitlab.test:8443, with KAS on https://gitlab.test:8443/-/kubernetes-agent, and the Kubernetes API proxy on https://gitlab.test:8443/-/kubernetes-agent/k8s-proxy.
The goal is to interact with the Kubernets API proxy locally using kubectl.

Steps:

Add a host alias in /etc/hosts for gitlab.test, pointing to your docker host address. This document uses a loopback alias 172.16.0.1.
Navigate to your Omnibus repository root and check out this branch.

Generate certificates:

/usr/local/opt/openssl/bin/openssl req -new -x509 \
    -days 365 \
    -newkey rsa:2048 \
    -nodes \
    -subj "/CN=gitlab.test/" \
    -addext "subjectAltName = DNS:gitlab.test" \
    -keyout "gitlab.test.key" \
    -out "gitlab.test.crt"

Run an Omnibus nightly, mounting the certs and relevant cookbooks, and forwarding the container port 443 to the host port 8443:

docker run \
    -v $(pwd)/gitlab.test.key:/etc/gitlab/ssl/gitlab.test.key \
    -v $(pwd)/gitlab.test.crt:/etc/gitlab/ssl/gitlab.test.crt \
    --privileged \
    -v $(pwd)/files/gitlab-cookbooks/gitlab:/opt/gitlab/embedded/cookbooks/gitlab \
    -v $(pwd)/files/gitlab-cookbooks/gitlab-kas:/opt/gitlab/embedded/cookbooks/gitlab-kas \
    -p 8443:443 \
    --rm -it \
    gitlab/gitlab-ee:nightly bash

Within the omnibus container:

# Add hosts alias to be locally reachable
echo '127.0.0.1 gitlab.test' >> /etc/hosts

# Set external URL
echo 'external_url "https://gitlab.test"' >> /etc/gitlab/gitlab.rb
echo "letsencrypt['enable'] = false" >> /etc/gitlab/gitlab.rb

# Enable kas
echo "gitlab_kas['enable'] = true" >> /etc/gitlab/gitlab.rb
echo "gitlab_kas['gitlab_ca_certificate_file'] = '/etc/gitlab/ssl/gitlab.test.crt'" >> /etc/gitlab/gitlab.rb

# Copy self-signed cert before reconfigure (so KAS trusts it)
cp /etc/gitlab/ssl/gitlab.test.crt /etc/gitlab/trusted-certs/

# Start runsv. Necessary because we are in a container; see https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/development/setup.md#setting-up-your-development-environment
/opt/gitlab/embedded/bin/runsvdir-start &

# Configure and start gitlab
gitlab-ctl reconfigure

From your local machine, create a cluster using kind with the following configuration:

# kind-loopback-alias.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
apiServerAddress: 0.0.0.0
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
version: v1beta2
kind: ClusterConfiguration
patch: |
    - op: add
    path: /apiServer/certSANs/-
    value: 172.16.0.1

Save the above as kind-loopback-alias.yaml and run the following commands:

# Create cluster
kind create cluster --config kind-loopback-alias.yaml

# Use loopback alias for API server address so we can call it from within an agentk container later on
sed -e s/0.0.0.0/172.16.0.1/ -i '' $HOME/.kube/config

From your browser, log into your omnibus instance at https://gitlab.test:8443
1. Create a project called root/gitlab-agent (arbitrary name),
2. Add a .gitlab.yml with a dummy job to enable CI:
```
# .gitlab-ci.yml
dummy-job: { script: echo }
```
3. Add a file .gitlab/agents/agent/config.yaml with the following content:
```
# .gitlab/agents/agent/config.yaml
observability:
logging:
    level: debug
```
4. Create an agent record in GitLab for the above configuration, but do not install it in your cluster. This can also work, but we need extra configuration for the certificates, so for the purpose of testing it is easier to run agentk directly.
  - To create an agent record, navigate to Infrastructure > Kubernetes > GitLab Agent managed clusters and click through
Save the token obtained from the last step into a file gitlab-agent-token in your Omnibus repo root (where you saved gitlab.test.crt).

Locally, run agentk:

docker run \
    --add-host gitlab.test:172.16.0.1 \
    -v $(pwd)/gitlab.test.crt:/etc/gitlab.test.crt \
    -v $(pwd)/gitlab-agent-token:/etc/gitlab-agent-token \
    -v $HOME/.kube/config:/etc/kubeconfig \
    -e KUBECONFIG=/etc/kubeconfig \
    --rm \
    registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:v14.4.0 \
        --kas-address=wss://gitlab.test:8443/-/kubernetes-agent/ \
        --token-file /etc/gitlab-agent-token \
        --ca-cert-file=/etc/gitlab.test.crt

Within the Omnibus container: In a rails console, switch the job status to running to enable that job's token, and grab its kubeconfig

# gitlab-rails console
b = Ci::Build.last
b.update(status: 'running')
puts ::Ci::GenerateKubeconfigService.new(b).execute.to_yaml

On your local machine, copy the kubeconfig YAML from above, and put it in a file $HOME/kas-kubeconfig.yaml. Run

sed -e s/gitlab.test/gitlab.test:8443/ -i '' $HOME/kas-kubeconfig.yaml
kubectl --kubeconfig $HOME/kas-kubeconfig --insecure-skip-tls-verify --context root/gitlab-agent:agent cluster-info

Edited Nov 01, 2021 by Hordur Freyr Yngvason

Add configuration for the KAS CI tunnel feature