Skip to content

Helm 3 Upgrade Fails with Immutable Field Error (spec.clusterIP)

Summary

Deployments using Helm 3.0.0 (with gitlab chart version 2.5.1) fail to deploy with spec.clusterIP: Invalid value: "": field is immutable error for the following services:

  • gitlab-exporter
  • gitlab-shell
  • gtilab-unicorn

After digging I found #1353 (closed) and helm#6378, but oddly the only template I could find that violated the "don't specify an clusterIP" rule was for gitaly. I did try using a modified version of the chart with the gitaly service template fixed, but as expected that had no effect here.

Steps to reproduce

Deploy a gitlab release with helm v3.0.0, then upgrade the release with any changes.

Configuration used

Note that we don't use the following optional sub-charts:

  • nginx-ingress
  • task-runner
  • certmanager
  • prometheus
  • registry

I think this is configuration-agnostic given the service manifests retrieved from the helm3 resource (see below).

Current behavior

Initial release of the gitlab chart with helm 3.0.0 works without a problem. Subsequent releases fail with:

FAILED RELEASES:
NAME
gitlab
in helmfile.d/02-internal-apps.gitlab.yaml: in .helmfiles[0]: in ../xetus-helmfiles/corp/gitlab.yaml: failed processing release gitlab: helm3 exited with status 1:
  client.go:399: Replaced "gitlab-postgresql" with kind ConfigMap for kind ConfigMap
  client.go:399: Replaced "gitlab-redis" with kind PersistentVolumeClaim for kind PersistentVolumeClaim
  client.go:399: Replaced "gitlab-gitlab-exporter" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-gitlab-shell" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-sidekiq-all-in-1" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-task-runner" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-unicorn" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-postgresql" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-redis" with kind Deployment for kind Deployment
  client.go:399: Replaced "gitlab-unicorn" with kind HorizontalPodAutoscaler for kind HorizontalPodAutoscaler
  client.go:399: Replaced "gitlab-gitaly" with kind StatefulSet for kind StatefulSet
  Error: UPGRADE FAILED: failed to replace object: Service "gitlab-gitlab-exporter" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "gitlab-gitlab-shell" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "gitlab-unicorn" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Expected behavior

It's possible there are two separate issues here:

  1. The gitlab helm chart can be released more than once with helm 3.0.0 without encountering a field is immutable error; and
  2. The gitaly chart's service template should not specify a clusterIP value.

Versions

  • Chart: 2.5.1
  • Platform:
    • Cloud: EKS
  • Kubernetes:
    • Client: v1.13.8
    • Server: v1.13.10-eks-5ac0f1
  • Helm:
    • Client: v3.0.0
    • Server:

Relevant Details

The service templates for the gitlab-exporter, gitlab-shell, and gitlab-unicorn charts clearly don't set a clusterIP value while the gitaly chart does. This is reflected in my deployment of the gitlab helm chart as confirmed by a helm get manifest call:

helm get manifest -n gitlab gitlab

<snip>
---
# Source: gitlab/charts/gitlab/charts/gitaly/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gitlab-gitaly
  namespace: gitlab
  labels:
    app: gitaly
    chart: gitaly-2.5.1
    release: gitlab
    heritage: Helm
    
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9236"
    
spec:
  type: ClusterIP
  clusterIP: "None"
  ports:
    - port: 8075
      name: gitaly
      targetPort: 8075
    - port: 9236
      name: gitaly-metrics
  selector:
    app: gitaly
    release: gitlab
---
# Source: gitlab/charts/gitlab/charts/gitlab-exporter/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gitlab-gitlab-exporter
  namespace: gitlab
  labels:
    app: gitlab-exporter
    chart: gitlab-exporter-2.5.1
    release: gitlab
    heritage: Helm
    
  annotations:
    
spec:
  type: ClusterIP
  ports:
    - port: 9168
      targetPort: 9168
      protocol: TCP
      name: gitlab-exporter
  selector:
    app: gitlab-exporter
    release: gitlab
---
# Source: gitlab/charts/gitlab/charts/gitlab-shell/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gitlab-gitlab-shell
  namespace: gitlab
  labels:
    app: gitlab-shell
    chart: gitlab-shell-2.5.1
    release: gitlab
    heritage: Helm
    
  annotations:
    
spec:
  type: ClusterIP
  ports:
    - port: 22
      targetPort: 2222
      protocol: TCP
      name: ssh
  selector:
    app: gitlab-shell
    release: gitlab
---
# Source: gitlab/charts/gitlab/charts/unicorn/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gitlab-unicorn
  namespace: gitlab
  labels:
    app: unicorn
    chart: unicorn-2.5.1
    release: gitlab
    heritage: Helm
    
  annotations:
    
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
      name: unicorn
    - port: 8181
      targetPort: 8181
      protocol: TCP
      name: workhorse
  selector:
    app: unicorn
    release: gitlab
---
<snip>
Edited by Jason Plum