Skip to content

Upgrade nginx controller to version >=0.26.0

Summary

We are seeing traffic imbalance on deployments with a large number of pods on GitLab.com which is due to a bug in nginx loadbalancing gitlab-com/gl-infra/delivery#1294 (comment 435870887).

The chart is not compatible with the more recent version, so we will need to make some updates.

https://github.com/kubernetes/ingress-nginx/issues/4296

0.26.0 is the first official release that has a fix for this, though there have been many other releases since then

https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md

With the current configuration, here are the errors we see when using 0.26.0

$ k logs gitlab-nginx-ingress-controller-7c97f5dfcf-csmgw -n gitlab
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       9ecec0de63451ef753d63fc20a1fb4cb97f876ca
  Build:         git-9ecec0de6
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: openresty/1.15.8.2

-------------------------------------------------------------------------------

Flag --force-namespace-isolation has been deprecated, This flag doesn't do anything.
I1026 09:27:33.720721       6 flags.go:194] Watching for Ingress class: gitlab-nginx
W1026 09:27:33.720774       6 flags.go:197] Only Ingresses with class "gitlab-nginx" will be processed by this Ingress controller
W1026 09:27:33.721106       6 flags.go:223] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W1026 09:27:33.721181       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1026 09:27:33.721430       6 main.go:181] Creating API client for https://10.97.0.1:443
I1026 09:27:33.731072       6 main.go:225] Running in Kubernetes cluster version v1.16+ (v1.16.13-gke.401) - git (clean) commit eb94c181eea5290e9da1238db02cfef263542f5f - platform linux/amd64
I1026 09:27:33.736770       6 main.go:89] Validated gitlab/gitlab-nginx-ingress-default-backend as the default backend.
I1026 09:27:33.986173       6 main.go:100] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I1026 09:27:34.016579       6 nginx.go:263] Starting NGINX Ingress controller
I1026 09:27:34.041198       6 event.go:255] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"gitlab", Name:"gitlab-nginx-ingress-tcp", UID:"ffc93449-412d-4c7f-a0bd-af9376c04d7e", APIVersion:"v1", ResourceVersion:"2762856", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap gitlab/gitlab-nginx-ingress-tcp
I1026 09:27:34.043394       6 event.go:255] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"gitlab", Name:"gitlab-nginx-ingress-controller", UID:"5cfb8e78-6f7e-459d-98df-1d6888b4cf91", APIVersion:"v1", ResourceVersion:"1331551", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap gitlab/gitlab-nginx-ingress-controller
E1026 09:27:35.120155       6 reflector.go:123] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:180: Failed to list *v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:gitlab:gitlab-nginx-ingress" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "gitlab"
E1026 09:27:36.122670       6 reflector.go:123] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:180: Failed to list *v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:gitlab:gitlab-nginx-ingress" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "gitlab"

(Summarize the bug encountered, concisely as possible)

Steps to reproduce

(Please provide the steps to reproduce the issue)

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

(Paste sanitized configuration here)

Current behavior

(What you're experiencing happening)

Expected behavior

(What you're expecting to happen)

Versions

  • Chart: (tagged version | branch | hash git rev-parse HEAD)
  • Platform:
    • Cloud: (GKE | AKS | EKS | ?)
    • Self-hosted: (OpenShift | Minikube | Rancher RKE | ?)
  • Kubernetes: (kubectl version)
    • Client:
    • Server:
  • Helm: (helm version)
    • Client:
    • Server:

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

Edited by John Jarvis