Skip to content

Workhorse Containers unable to start

Summary

After an attempt to upgrade the helm chart, wokrhorse is unable to start.

  gitlab-workhorse:
    Container ID:   containerd://cdae5671f4485083fe8c4f394301d638729b40eaae59e50bf6dc5a765994c200
    Image:          dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-workhorse-ee:14-5-202111030920-f39fff7ba6a
    Image ID:       dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-workhorse-ee@sha256:45a5b4370f5ef9c51f5fe6816f2b49d07e2b6ec8d3967c4f69a602c15eac9631
    Port:           8181/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 03 Nov 2021 09:20:39 -0400
      Finished:     Wed, 03 Nov 2021 09:20:39 -0400
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    28m                    default-scheduler  Successfully assigned gitlab-cny/gitlab-cny-webservice-git-d64f959fc-g9pfz to gke-gstg-gitlab-gke-git-https-0-e2b2f828-ivqn
  Warning  FailedMount  28m                    kubelet            MountVolume.SetUp failed for volume "workhorse-config" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulling      28m                    kubelet            Pulling image "busybox:latest"
  Normal   Pulled       28m                    kubelet            Container image "dev.gitlab.org:5005/gitlab/charts/components/images/alpine-certificates:20191127-r2" already present on machine
  Normal   Created      28m                    kubelet            Created container write-instance-name
  Normal   Started      28m                    kubelet            Started container write-instance-name
  Normal   Pulled       28m                    kubelet            Successfully pulled image "busybox:latest" in 130.030617ms
  Normal   Created      28m                    kubelet            Created container certificates
  Normal   Started      28m                    kubelet            Started container certificates
  Normal   Pulling      28m                    kubelet            Pulling image "busybox:latest"
  Normal   Pulled       28m                    kubelet            Successfully pulled image "busybox:latest" in 113.847372ms
  Normal   Created      28m                    kubelet            Created container configure
  Normal   Started      28m                    kubelet            Started container configure
  Normal   Pulling      28m                    kubelet            Pulling image "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-webservice-ee:14-5-202111030920-f39fff7ba6a"
  Normal   Pulled       27m                    kubelet            Successfully pulled image "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-webservice-ee:14-5-202111030920-f39fff7ba6a" in 28.600678454s
  Normal   Created      27m                    kubelet            Created container dependencies
  Normal   Started      27m                    kubelet            Started container dependencies
  Normal   Pulled       27m                    kubelet            Container image "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-webservice-ee:14-5-202111030920-f39fff7ba6a" already present on machine
  Normal   Created      27m                    kubelet            Created container webservice
  Normal   Started      27m                    kubelet            Started container webservice
  Normal   Pulling      27m                    kubelet            Pulling image "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-workhorse-ee:14-5-202111030920-f39fff7ba6a"
  Normal   Pulled       27m                    kubelet            Successfully pulled image "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-workhorse-ee:14-5-202111030920-f39fff7ba6a" in 7.87330692s
  Normal   Created      27m (x2 over 27m)      kubelet            Created container gitlab-workhorse
  Normal   Started      27m                    kubelet            Started container gitlab-workhorse
  Normal   Pulled       27m                    kubelet            Container image "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-workhorse-ee:14-5-202111030920-f39fff7ba6a" already present on machine
  Warning  BackOff      3m18s (x118 over 27m)  kubelet            Back-off restarting failed container

I think the failed mount was temporary and may not be the root cause.

% k logs gitlab-cny-webservice-git-d64f959fc-g9pfz -n gitlab-cny -c gitlab-workhorse
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb templates from /var/opt/gitlab/templates
Begin parsing .tpl templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/workhorse-config.toml
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
+ exec /scripts/exec-env /bin/sh -c /scripts/start-workhorse
configFile: Near line 3 (last key parsed ''): expected a top-level item to end with a newline, comment, or EOF, but got '[' instead

Steps to reproduce

We attempted to set a sleep on this deployment to gather the constructed file, however, this file did not get created.

% kubectl exec -it gitlab-cny-webservice-api-856c9c8457-7j79r -n gitlab-cny -c gitlab-workhorse -- /bin/sh
$  sh -x /config-webservice/configure ; sh -x /config-workhorse/configure ; mkdir -p -m 3770 /tmp/gitlab
/bin/sh: 3: -p: not found
sh: 0: Can't open /config-workhorse/configure
$ /config-webservice/configure
/bin/sh: 4: /config-webservice/configure: not found
$ cat /config-webservice/configure
cat: /config-webservice/configure: No such file or directory
$ cd /
$ ls
bin  boot  build-scripts  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  scripts  srv  sys  tmp  usr  var

Current behavior

Workhorse container is Crashing immediately

Expected behavior

Workhorse is able to start

Versions

  • Chart: e5cdb606991fb80833818735a22c56af888131c0
  • Platform:
    • Cloud: GKE
  • Kubernetes:
    • Client: 1.20.9
    • Server: 1.20.10-gke.1600
  • Helm: 3.7.1