pg-upgrade broken for Geo secondaries
Note: This issue is about single-node Geo installations. Multi-node ones are already handled by #6756 (closed)
!5728 (merged) inadvertently broke pg-upgrade
for Geo secondaries. When running pg-upgrade on Geo secondaries, the reconfigure run as part of upgrading regular DB writes a VERSION
file in the Geo PG directory using the psql --version
command (with the new binaries), and this will cause a Geo PG restart. However, since the Geo PG data is of old PG version, this restart will fail, eventually causing the pg-upgrade
to fail.
To reproduce
-
Deploy Geo instance using GitLab 14.10.3 and get it upgraded to GitLab 15.0. This is to have a Geo instance which still runs PG 12. Else, you can use a package from https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/239480 which is 15.0.0 but has PG 12 as default for fresh installs, and deploy Geo using it.
-
Upgrade PG using
sudo gitlab-ctl pg-upgrade -V 13
-
See it fail with the following log
* version_file[Create version file for PostgreSQL] action create * file[/var/opt/gitlab/geo-postgresql/VERSION] action create - update content in file /var/opt/gitlab/geo-postgresql/VERSION from 7fc47b to 45ab29 --- /var/opt/gitlab/geo-postgresql/VERSION 2022-05-27 07:33:28.541510700 +0000 +++ /var/opt/gitlab/geo-postgresql/.chef-VERSION20220527-26681-l64ykl 2022-05-27 07:36:40.169720441 +0000 @@ -1,2 +1,2 @@ -postgres (PostgreSQL) 12.10 +postgres (PostgreSQL) 13.6 Recipe: postgresql::standalone * runit_service[postgresql] action restart (up to date) Recipe: gitlab-ee::geo-postgresql * runit_service[geo-postgresql] action restart ================================================================================ Error executing action `restart` on resource 'runit_service[geo-postgresql]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/geo-postgresql ---- STDOUT: timeout: down: /opt/gitlab/service/geo-postgresql: 0s, normally up, want up STDERR: ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/geo-postgresql ---- Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/geo-postgresql returned 1
-
Check logs of
geo-postgresql
service and see the following2022-05-27_07:33:29.05340 LOG: starting PostgreSQL 12.10 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit 2022-05-27_07:33:29.05591 LOG: listening on Unix socket "/var/opt/gitlab/geo-postgresql/.s.PGSQL.5431" 2022-05-27_07:33:29.12769 LOG: database system was shut down at 2022-05-27 07:33:29 GMT 2022-05-27_07:33:29.13320 LOG: database system is ready to accept connections 2022-05-27_07:36:39.92387 LOG: no match in usermap "gitlab" for user "gitlab_geo" authenticated as "root" 2022-05-27_07:36:39.92392 FATAL: Peer authentication failed for user "gitlab_geo" 2022-05-27_07:36:39.92393 DETAIL: Connection matched pg_hba.conf line 70: "local all all peer map=gitlab" 2022-05-27_07:36:40.62977 received TERM from runit, sending INT instead to force quit connections 2022-05-27_07:36:40.63206 LOG: received fast shutdown request 2022-05-27_07:36:40.63677 LOG: aborting any active transactions 2022-05-27_07:36:40.63690 LOG: background worker "logical replication launcher" (PID 25692) exited with exit code 1 2022-05-27_07:36:40.63858 LOG: shutting down 2022-05-27_07:36:40.67215 LOG: database system is shut down 2022-05-27_07:36:40.67753 2022-05-27 07:36:40.677 GMT [27121] LOG: unrecognized configuration parameter "wal_keep_segments" in file "/var/opt/gitlab/geo-postgresql/data/runtime.conf" line 30 2022-05-27_07:36:40.67759 2022-05-27 07:36:40.677 GMT [27121] FATAL: configuration file "/var/opt/gitlab/geo-postgresql/data/runtime.conf" contains errors
This is because the binaries were updated by pg-upgrade
as part of upgrading the regular PG, but the change to VERSION
file caused a restart of Geo PG whose configuration was still for PG 12.