Include example configuration for using KAS with AWS ALB
Summary
I worked on installing GitLab Helm chart on EKS cluster and using AWS ALB (application load balancer) as my load balancer. Layer 7 example in the AWS examples did not work. The example is using the chart default nginx-ingress
controller. Many AWS users disable the nginx-ingress
and use the AWS Load Balancer Controller. In GitLab docs there are no configuration examples for KAS to work with ALB controller. It was difficult and time consuming to get it to work as ALB wants some specific annotations. We should include examples and/or notes for this for the growing number of customers using EKS and ALB. A working configuration is included in this issue as screenshots and kas-ingress.yaml.
This issue is to include the findings here in an appropriate place, either in GitLab docs and/or in the chart examples.
Findings
-
Required Ingress annotations:
- Full list of annotations:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/group.name: gitlab-alb
alb.ingress.kubernetes.io/healthcheck-path: /liveness
alb.ingress.kubernetes.io/healthcheck-port: "8151"
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
#alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
#alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/certificate-arn: <CERTIFICATE_ARN>
alb.ingress.kubernetes.io/tags: Name=gitlab-alb,Environment=test
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=4000,routing.http2.enabled=false
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=86400
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/tls-acme: "true"
- list of annotations that are critical for ALB to work:
alb.ingress.kubernetes.io/healthcheck-path: /liveness
alb.ingress.kubernetes.io/healthcheck-port: "8151"
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=4000,routing.http2.enabled=false
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=86400
alb.ingress.kubernetes.io/target-type: ip
- The kas Ingress rules in our templates cause ALB to not route KAS requests properly. This is due to how ALB Listener's rules are checked. The rules are saved in the order they are defined in
spec.rules
. If Rule-1 matches the path, then Rule-2 is not checked. In other words it does not work like Nginx's longest pattern match. In order to route KAS requests properly, in the Ingress manifest, rules must be in the order where path/
(/*
for alb) comes last. Note: This is the reason I disabled Ingress for kas in myvalues.yaml
and used an external kas manifest (kas-ingress.yaml attached).
spec:
ingressClassName: alb
rules:
- host: kas.supermunn.com
http:
paths:
- backend:
service:
name: gitlab-kas
port:
number: 8154
path: /k8s-proxy/
pathType: Prefix
- backend:
service:
name: gitlab-kas
port:
number: 8150
path: /
pathType: Prefix
- In GitLab Helm chart
values.yaml
, kas service type needs to beNodePort
and annotations for alb must be added.
kas:
enabled: true
service:
type: NodePort
apiExternalPort: 8153 # port for connections from the GitLab backend
externalPort: 8150
internalPort: 8150
apiInternalPort: 8153
kubernetesApiPort: 8154
privateApiPort: 8155
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:<AWS_REGION>:<AWS_ACCOUNT_ID>:certificate/<ACM_CERTIFICATE_ID>
# Configure which ports are to terminate SSL.
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
#service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
#targetPorts:
# https: http # the ELB will send HTTP to 443
Versions
-
Chart: (tagged version | branch | hash
git rev-parse HEAD
) -
Platform:
- Cloud: EKS
-
Kubernetes: (
kubectl version
)- Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:31:32Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/amd64"}
- Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
-
Helm: (
helm version
) version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"dirty", GoVersion:"go1.16.4"}