Initd/systemd script is waiting for unicorn pid file even if puma is selected
Summary
In 12.9.2 (source installation) when Puma is used as default webserver, the init.d script is still waiting for the unicorn.pid file to be written, causing startup to fail/timeout.
Steps to reproduce
Ensure USE_UNICORN=""
in /etc/init.d/gitlab and
run systemctl start gitlab
What is the current bug behavior?
Gitlab startup fails after 30 seconds, waiting for the incorrect pid file to be written.
What is the expected correct behavior?
Gitlab should startup, and the init-script should wait for the correct pid file.
Relevant logs and/or screenshots
Apr 06 10:34:54 share gitlab[101653]: Starting GitLab web server (puma)
Apr 06 10:34:55 share gitlab[101653]: {"timestamp":"2020-04-06T08:34:55.241Z","pid":101677,"message":"Puma starting in cluster mode..."}
Apr 06 10:34:55 share gitlab[101653]: {"timestamp":"2020-04-06T08:34:55.241Z","pid":101677,"message":"* Version 4.3.3.gitlab.2 (ruby 2.6.5-p114), codename: Mysterious Traveller"}
Apr 06 10:34:55 share gitlab[101653]: {"timestamp":"2020-04-06T08:34:55.241Z","pid":101677,"message":"* Min threads: 1, max threads: 16"}
Apr 06 10:34:55 share gitlab[101653]: {"timestamp":"2020-04-06T08:34:55.241Z","pid":101677,"message":"* Environment: production"}
Apr 06 10:34:55 share gitlab[101653]: {"timestamp":"2020-04-06T08:34:55.241Z","pid":101677,"message":"* Process workers: 4"}
Apr 06 10:34:55 share gitlab[101653]: {"timestamp":"2020-04-06T08:34:55.241Z","pid":101677,"message":"* Preloading application"}
Apr 06 10:35:33 share gitlab[101653]: {"timestamp":"2020-04-06T08:35:33.831Z","pid":101677,"message":"* Listening on unix:///home/git/gitlab/tmp/sockets/gitlab.socket"}
Apr 06 10:35:33 share gitlab[101653]: {"timestamp":"2020-04-06T08:35:33.831Z","pid":101677,"message":"! WARNING: Detected 2 Thread(s) started in app boot:"}
Apr 06 10:35:33 share gitlab[101653]: {"timestamp":"2020-04-06T08:35:33.831Z","pid":101677,"message":"! #\u003cThread:0x0000000009aae660@/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:336 sleep\u003e - /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:339:in `sleep'"}
Apr 06 10:35:33 share gitlab[101653]: {"timestamp":"2020-04-06T08:35:33.832Z","pid":101677,"message":"! #\u003cThread:0x000000000a9ebf60@/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.1/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-timeout-0.5.1/lib/rack/timeout/support/scheduler.rb:91:in `sleep'"}
Apr 06 10:35:33 share gitlab[101653]: {"timestamp":"2020-04-06T08:35:33.832Z","pid":101677,"message":"* Daemonizing..."}
Apr 06 10:35:33 share gitlab[101653]: The Sidekiq job dispatcher is already running with pid 101190, not restarting
Apr 06 10:35:33 share gitlab[101653]: The GitLab Workhorse is already running with pid 101116, not restarting
Apr 06 10:35:33 share gitlab[101653]: Gitaly is already running with pid 101115, not restarting
Apr 06 10:36:04 share gitlab[101653]: ..............................Waited 30s for the processes to write their pids, something probably went wrong.
Apr 06 10:36:04 share systemd[1]: gitlab.service: Control process exited, code=exited status=1
Apr 06 10:36:04 share systemd[1]: Failed to start LSB: GitLab git repository management.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 3.0.3 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.0.6 Git Version: 2.26.0 Sidekiq Version:5.2.7 Go Version: go1.13.5 linux/amd64 GitLab information Version: 12.9.2 Revision: ac5568eb5d8 Directory: /home/git/gitlab DB Adapter: PostgreSQL DB Version: 9.6.17 URL: https://share.nbl.nislab.no HTTP Clone URL: https://share.nbl.nislab.no/some-group/some-project.git SSH Clone URL: git@share.nbl.nislab.no:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 12.0.0 Repository storage paths: - default: /home/git/repositories GitLab Shell path: /home/git/gitlab-shell Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0) Running /home/git/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... no Try fixing it: Re-download the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Projects have namespace: ... 1/11 ... yes 15/13 ... yes 50/15 ... yes 31/18 ... yes 49/20 ... yes 57/21 ... yes 57/22 ... yes 57/23 ... yes 57/24 ... yes 57/25 ... yes 57/26 ... yes 59/29 ... yes 52/30 ... yes 50/31 ... yes 50/32 ... yes 50/34 ... yes 30/35 ... yes 53/36 ... yes 1/38 ... yes 53/39 ... yes 53/40 ... yes 41/42 ... yes 38/43 ... yes 39/44 ... yes 80/46 ... yes 37/47 ... yes 37/48 ... yes 37/49 ... yes 37/50 ... yes 37/51 ... yes 59/52 ... yes 82/53 ... yes 36/54 ... yes 32/55 ... yes 37/56 ... yes 84/57 ... yes 15/60 ... yes 36/61 ... yes 29/63 ... yes 1/65 ... yes 33/68 ... yes 29/69 ... yes 97/72 ... yes 43/73 ... yes 59/74 ... yes 54/75 ... yes 37/76 ... yes 59/77 ... yes 99/78 ... yes 99/79 ... yes 99/80 ... yes 97/81 ... yes 97/83 ... yes 84/85 ... yes 59/86 ... yes 56/88 ... yes 40/89 ... yes 102/90 ... yes 97/91 ... yes 97/92 ... yes 74/93 ... yes 114/94 ... yes 105/95 ... yes 56/96 ... yes 103/98 ... yes 102/99 ... yes 115/100 ... yes 102/101 ... yes 102/103 ... yes 37/104 ... yes 103/105 ... yes 59/106 ... yes 103/107 ... yes 34/108 ... yes 56/109 ... yes 121/110 ... yes 52/111 ... yes 74/112 ... yes 117/114 ... yes 53/115 ... yes 53/116 ... yes 115/117 ... yes 37/118 ... yes 51/119 ... yes 129/120 ... yes 74/121 ... yes 15/122 ... yes 52/123 ... yes 126/124 ... yes 15/125 ... yes 49/126 ... yes 37/128 ... yes 115/129 ... yes 15/130 ... yes 132/132 ... yes 115/133 ... yes 115/135 ... yes 138/136 ... yes 76/138 ... yes 44/139 ... yes 140/140 ... yes 37/141 ... yes 37/142 ... yes 32/143 ... yes 141/145 ... yes 57/146 ... yes 54/147 ... yes 73/148 ... yes 77/151 ... yes 103/152 ... yes 152/153 ... yes 140/154 ... yes 140/157 ... yes 54/158 ... yes 54/159 ... yes 54/160 ... yes 54/161 ... yes 153/162 ... yes 73/163 ... yes 132/164 ... yes 54/165 ... yes 73/166 ... yes 75/167 ... yes 73/172 ... yes 54/173 ... yes 141/174 ... yes 141/175 ... yes 73/176 ... yes 73/177 ... yes 144/180 ... yes 144/181 ... yes 153/182 ... yes 41/183 ... yes 41/184 ... yes 76/186 ... yes 140/188 ... yes 153/189 ... yes 83/190 ... yes 73/191 ... yes 144/193 ... yes 144/194 ... yes 164/195 ... yes 102/196 ... yes 158/198 ... yes 165/199 ... yes 165/200 ... yes 43/201 ... yes 166/202 ... yes 144/203 ... yes 166/204 ... yes 103/205 ... yes 41/206 ... yes 41/207 ... yes 161/208 ... yes 73/210 ... yes 168/211 ... yes 172/212 ... yes 152/213 ... yes 174/214 ... yes 173/215 ... yes 175/216 ... yes 103/217 ... yes 152/218 ... yes 152/219 ... yes 152/220 ... yes 152/221 ... yes 152/222 ... yes 152/223 ... yes 152/224 ... yes 152/225 ... yes 152/226 ... yes 152/227 ... yes 165/228 ... yes 36/229 ... yes 152/230 ... yes 152/231 ... yes 172/232 ... yes 172/233 ... yes 54/234 ... yes 38/235 ... yes 161/237 ... yes 172/238 ... yes 144/239 ... yes 38/240 ... yes 38/241 ... yes 38/242 ... yes 38/243 ... yes 38/244 ... yes 38/245 ... yes 38/246 ... yes 180/248 ... yes 75/249 ... yes 37/250 ... yes 37/251 ... yes 36/252 ... yes 172/253 ... yes 73/254 ... yes 183/256 ... yes 75/257 ... yes 126/258 ... yes 162/259 ... yes 172/260 ... yes 129/262 ... yes 126/263 ... yes 186/264 ... yes 172/265 ... yes 83/266 ... yes 83/267 ... yes 152/268 ... yes 15/269 ... yes 140/270 ... yes 172/271 ... yes 172/272 ... yes 38/273 ... yes 153/274 ... yes 153/275 ... yes 153/277 ... yes 54/278 ... yes 187/279 ... yes 37/280 ... yes 37/281 ... yes 172/282 ... yes 172/283 ... yes 172/284 ... yes 54/285 ... yes 144/286 ... yes 189/287 ... yes 186/288 ... yes 186/289 ... yes 37/290 ... yes 172/291 ... yes 189/292 ... yes 126/293 ... yes 38/294 ... yes 38/295 ... yes 38/296 ... yes 140/297 ... yes 195/298 ... yes 38/299 ... yes 195/300 ... yes 196/301 ... yes 172/302 ... yes 195/303 ... yes 132/304 ... yes 195/306 ... yes 196/307 ... yes 82/308 ... yes 103/309 ... yes 153/310 ... yes 172/311 ... yes 49/312 ... yes 166/313 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.26.0) Git user has default SSH configuration? ... yes Active users: ... 59 Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Please note that the init-script test above obviously failed becasue I have edited it locally to fix this temporarily.
This: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/support/init.d/gitlab#L39 should be set according to which webserver is selected. Maybe with the same logic as here? https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/support/init.d/gitlab#L71-75
I hot-patched it in my environment with web_server_pid_path="$pid_path/$use_web_server.pid"
added after line 71-75 in the aforementioned code links.