Add zoekt-gateway nginx container with optional TLS
Related to gitlab-org/gitlab#389749 (closed) . We need to add encryption and authentication to Zoekt. Zoekt is a HTTP server called from gitlab-rails. This MR introduces the encryption part with the following changes:
- An nginx gateway in a container that proxies requests to
zoekt-webserver
- An optional configuration
gateway.tls.certificate.enabled
which will mount a namedCertificate
for nginx to use - An optional configuration
gateway.tls.certificate.create
which will create aCertificate
using a named certmanagerIssuer
- Change the service to expose the gateway as well
These were extracted from this larger MR !14 (merged) which also had basicAuth. But the basicAuth was proving to be complicated so I wanted to separate the changes.
There are lots of configuration options available here but the idea is to give flexibility as we may have scenarios where we want the certificate created in the chart and may later change our mind and want the certificate created in the parent chart or maybe outside the charts altogether depending on what issuer we use.
I did not proxy requests to the indexserver for now as this is being re-written in gitlab-org/gitlab#402984 (closed) . Later on this gateway will likely proxy to the indexserver as well. I also don't want to remove the webserver
service yet as this allows us to roll this out in a controlled way and delete the webserver later once we've migrated callers to the gateway.
Why not just use TLS built into Ingress?
I realise this could also be implemented in an Ingress
or Service
but it didn't solve our requirements for a few reasons:
- This doesn't actually encrypt traffic all the way to the pod and therefore there is still unencrypted network traffic in the cluster
- If we added basicAuth at an Ingress/Service level then the pods were still open to receive traffic from anywhere in the cluster. This was specifically a concern for infrasec because our NetworkPolicy is so broad in our deployments which means we basically allow anyone in the cluster to talk to any other unauthenticated service which allows escalating access more easily
- This gateway implementation will likely become more GitLab specific over time and may end up being written in Go with custom logic. At present Zoekt is a very simple single server solution but over time we'll need to build sharding and replication into Zoekt and we'll need an abstraction layer somewhere for this and a gateway in front of the lower level Zoekt binaries could serve this purpose
How to test
How to test
- Install cert manager in your cluster
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml
- Create a config file
cabootstrap.yaml
that will be used to bootstrap a CA on your cluster with the following contentapiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: my-selfsigned-ca namespace: default spec: isCA: true commonName: my-selfsigned-ca secretName: root-secret privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned-issuer kind: ClusterIssuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: my-ca-issuer namespace: default spec: ca: secretName: root-secret
- Apply this config:
kubectl apply -f cabootstrap.yaml
- Install the chart and have it create a certificate:
helm install --set gateway.tls.certificate.enabled=true --set gateway.tls.certificate.create=true --set gateway.tls.certificate.issuer.name=my-ca-issuer --set gateway.tls.certificate.dnsNames='{zoekt.example.com}' gitlab-zoekt .
- Curl the API and confirm it uses TLS:
kubectl exec gitlab-zoekt-0 -- curl -XPOST -d '{"Q":"gitaly"}' 'https://127.0.0.1:8080/api/search' -i -k
- View nginx logs
kubectl logs gitlab-zoekt-0 zoekt-gateway