Reconfigure appears to not start Patroni (and Postgres) properly
While reconfiguring a node after dealing with #5746 (closed) I got a timeout start error for Postgres via Patroni:
* ruby_block[wait for postgresql to start] action run
================================================================================
Error executing action `run` on resource 'ruby_block[wait for postgresql to start]'
================================================================================
Timeout::Error
--------------
execution expired
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:95:in `sleep'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:95:in `block (3 levels) in from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:94:in `block (2 levels) in from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb
92: ruby_block 'wait for postgresql to start' do
93: block do
94: Timeout.timeout(30) do
95: sleep 2 until pg_helper.ready?
96: end
97: end
98: end
99:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:92:in `from_file'
ruby_block("wait for postgresql to start") do
action [:run]
default_guard_interpreter :default
declared_type :ruby_block
cookbook_name "patroni"
recipe_name "enable"
block #<Proc:0x000055e67361d1e0 /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:93>
block_name "wait for postgresql to start"
end
System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=18.04
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
execution expired
On further investigation it appears that Patroni doesn't seem to be started a part of gitlab-ctl reconfigure
(on the latest nightly omnibus package - 13.6.0-pre (400d0718b06)
) after it was specifically stopped with gitlab-ctl stop
. Starting it again with gitlab-ctl start
beforehand and then re-running reconfigure solved the issue.
It wouldn't be too uncommon for users to have patroni stopped beforehand before running reconfigure so this should be addressed.
Edited by Grant Young