KAS pod timeout trying to connect to itself when TLS enabled for private API
The problem
When using the CI/CD tunnel to reach the Kubernetes API, KAS pods fails while trying to talk to each other.
This only happens when TLS is enabled for the KAS Private API.
Logs
KAS log failure.
It happens since !788 (merged).
kas {"level":"error","time":"2022-12-23T21:58:42.246Z","msg":"Failed to open new stream to kas","grpc_service":"gitlab.agent.kubernetes_api.rpc.KubernetesApi","grpc_method":"MakeRequest","agent_id":1,"kas_url": │
│ "grpcs://10.56.2.30:8155","error":"rpc error: code = Unavailable desc = io: read/write on closed pipe"} │
│ kas {"level":"error","time":"2022-12-23T21:58:42.802Z","msg":"Failed to open new stream to kas","grpc_service":"gitlab.agent.kubernetes_api.rpc.KubernetesApi","grpc_method":"MakeRequest","agent_id":1,"kas_url": │
│ "grpcs://10.56.0.21:8155","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.56.0.21:8155: i/o timeout\""}
The 10.56.0.21
IP that we see in the logs, is the IP of the pod that I took the logs from. So, it looks like a timeout while trying to open a connection to itself.
Reproducing with a GitLab chart
Follow the steps to enable TLS for KAS. Then try to use the tunnel.
I've built KAS image tags for the first version which fails:
- commit:
ee267c52271d153b93360488feaefea7e66f0441
- image tag:
registry.gitlab.com/gitlab-org/build/cng/gitlab-kas:6d72fed675cf4590c48eec8e9f761da054e80588
And the last successful commit:
- commit:
af54e64efc672b0d5c1d2ac9bb0757c39871ca23
- tag:
registry.gitlab.com/gitlab-org/build/cng/gitlab-kas:92cbdb9e7a7f7b27e50ad44e578e87598759a57a
To choose the KAS deployed version you can override gitlab.kas.image.tag: XYZ
and gitlab.kas.image.pullPolicy: Always
Helm value.
Reproducing with GDK
KAS Config used
# My KAS config:
agent:
listen:
network: "tcp"
address: "172.16.123.1:8159"
websocket: true
kubernetes_api:
listen:
network: "tcp"
address: "172.16.123.1:8154"
url_path_prefix: "/-/k8s-proxy/"
gitlab:
address: "https://gdk.test:3333"
authentication_secret_file: "/Users/my-user/dev/gitlab-development-kit/gitlab/.gitlab_kas_secret"
ca_certificate_file: "/Users/my-user/dev/gitlab-development-kit/localhost.crt"
api:
listen:
network: "tcp"
address: "172.16.123.1:8153"
authentication_secret_file: "/Users/my-user/dev/gitlab-development-kit/gitlab/.gitlab_kas_secret"
redis:
network: unix
server:
address: "/Users/my-user/dev/gitlab-development-kit/redis/redis.socket"
private_api:
listen:
network: "tcp"
address: "172.16.123.1:8155"
authentication_secret_file: "/Users/my-user/dev/gitlab-development-kit/gitlab/.gitlab_kas_secret"
certificate_file: "/Users/my-user/dev/gitlab-development-kit/gdk.test.crt"
key_file: "/Users/my-user/dev/gitlab-development-kit/gdk.test-key.key"
# ca_certificate_file: "/Users/my-user/Library/Application\ Support/mkcert/rootCA.pem"
observability:
logging:
level: debug
Environment Variables used
For my own GDK setup, I've set the following envs:
host=gdk.test
OWN_PRIVATE_API_HOST=gdk.test
OWN_PRIVATE_API_URL=grpcs://172.16.123.1:8155
port=3443
Edited by João Alexandre Cunha