puma['socket'] configuration option needs to also reconfigure workhorse
Summary
Investigation on a customer call for a ticket (internal links) identified that there's a step missing in gitlab-ctl reconfigure
to modify /opt/gitlab/sv/gitlab-workhorse/run
when the puma socket is moved to a non-default location.
Steps to reproduce
- Vanilla Omnibus 13.x install with (default) Puma as Rails server.
- create alternative Puma socket location
mkdir -p /var/tmp/gitlab-rails/sockets/
chown git:gitlab-www /var/tmp/gitlab-rails/sockets/
chmod 750 /var/tmp/gitlab-rails/sockets/
- reconfigure Omnibus
sed -i "s~^# puma\['socket'\] = .*\$~puma\['socket'\] = '/var/tmp/gitlab-rails/sockets/gitlab.socket'~g" /etc/gitlab/gitlab.rb
gitlab-ctl reconfigure
gitlab-ctl restart
What is the current bug behavior?
Puma relocates its socket, workhorse innocently carries on looking in the default location.
Gitlab doesn't work, returning 502. "GET / HTTP/2.0" 502
workhorse log:
{"correlation_id":"DBE9J4yBBV2","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"error","time":"2020-08-07T16:55:51+01:00","uri":"/"}
What is the expected correct behavior?
Both Puma and Workhorse are reconfigured to use a socket in a different location.
Relevant logs
Here's a comparison of what happened with Unicorn so Workhorse could locate the socket, vs. Puma.
-# unicorn['socket'] = '/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
+unicorn['socket'] = '/var/tmp/gitlab-rails/sockets/gitlab.socket'
chef output - unicorn / 12.10
* directory[/var/tmp/gitlab-rails/sockets] action create
- change mode from '0755' to '0750'
- restore selinux security context
* directory[/var/opt/gitlab/gitlab-rails/etc] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/unicorn.rb] action create
- update content in file /var/opt/gitlab/gitlab-rails/etc/unicorn.rb from 7b049f to 86f257
--- /var/opt/gitlab/gitlab-rails/etc/unicorn.rb 2020-03-02 19:50:18.227549365 +0000
+++ /var/opt/gitlab/gitlab-rails/etc/.chef-unicorn20200807-24894-vnofz7.rb 2020-08-07 15:21:17.570103594 +0000
@@ -5,7 +5,7 @@
# What ports/sockets to listen on, and what options for them.
listen "127.0.0.1:8080", :tcp_nopush => true
-listen "/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket", :backlog => 1024
+listen "/var/tmp/gitlab-rails/sockets/gitlab.socket", :backlog => 1024
working_directory '/var/opt/gitlab/gitlab-rails/working'
- restore selinux security context
[..]
* template[/opt/gitlab/sv/gitlab-workhorse/run] action create
- update content in file /opt/gitlab/sv/gitlab-workhorse/run from a979e8 to 69c765
--- /opt/gitlab/sv/gitlab-workhorse/run 2019-10-08 03:36:27.450422836 +0000
+++ /opt/gitlab/sv/gitlab-workhorse/.chef-run20200807-24894-hrcwhc 2020-08-07 15:21:17.798123246 +0000
@@ -16,7 +16,7 @@
-listenUmask 0 \
-listenAddr /var/opt/gitlab/gitlab-workhorse/socket \
-authBackend http://localhost:8080 \
- -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket \
+ -authSocket /var/tmp/gitlab-rails/sockets/gitlab.socket \
-documentRoot /opt/gitlab/embedded/service/gitlab-rails/public \
-pprofListenAddr ''\
-prometheusListenAddr localhost:9229 \
- restore selinux security context
-# puma['socket'] = '/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
+puma['socket'] = '/var/tmp/gitlab-rails/sockets/gitlab.socket'
chef output - puma / 13.2
* puma_config[/var/opt/gitlab/gitlab-rails/etc/puma.rb] action create
* directory[/var/opt/gitlab/gitlab-rails/etc] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/puma.rb] action create
- update content in file /var/opt/gitlab/gitlab-rails/etc/puma.rb from bd681a to b592b9
--- /var/opt/gitlab/gitlab-rails/etc/puma.rb 2020-08-07 16:34:58.462988950 +0100
+++ /var/opt/gitlab/gitlab-rails/etc/.chef-puma20200807-15734-1nr91rl.rb 2020-08-07 16:54:25.132720998 +0100
@@ -34,7 +34,7 @@
# Bind the server to "url". "tcp://", "unix://" and "ssl://" are the only
# accepted protocols.
-bind 'unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
+bind 'unix:///var/tmp/gitlab-rails/sockets/gitlab.socket'
bind 'tcp://127.0.0.1:8080'
- restore selinux security context
Details of package version
reproduced on 13.1.4, 13.2.3
Provide the package version installation details
gitlab-ee-13.2.3-ee.0.el7.x86_64
Environment details
- Operating System: Centos7
- Installation Target, remove incorrect values:
- VM: KVM
- Installation Type, remove incorrect values:
- New Installation @13.0.0
- Upgrade from version [..] 13.1.0, 13.1.4, 13.2.3
- Is there any other software running on the machine: gitlab-runner
- Is this a single or multiple node installation? single
- Resources
- CPU: 2 vcpu
- Memory total: 6gb
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
puma['socket'] = '/var/tmp/gitlab-rails/sockets/gitlab.socket'
Edited by Ben Prescott_