Geo: Improve docs to help with PG upgrades
What does this MR do?
Follow up items that will help Geo HA customers found during omnibus-gitlab#4975 (comment 319485449):
Modify https://docs.gitlab.com/ee/administration/geo/replication/high_availability.html:
-
Recommend disabling more services when configuring the secondary’s read-replica node (esp. consul and/or repmgr) because the pg-upgrade
command “Detected an HA cluster” and errors
Modify https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html:
-
Add the following error for those that accidentally have consul and/or repmgr services configured on the read-replica node
mkozono-ansible@mkozono-ha-omnibus4838-secondary-geo:~$ sudo gitlab-ctl pg-upgrade --target-version=11
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 11.7
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details
Please hit Ctrl-C now if you want to cancel the operation.
..............................Detected an HA cluster.
Error running command: /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul
ERROR: psql: FATAL: role "gitlab-consul" does not exist
Traceback (most recent call last):
10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:171:in `block in load_file'
4: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:248:in `is_master?'
3: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:100:in `execute_psql'
2: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:113:in `cmd'
1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ----
STDOUT:
STDERR: psql: FATAL: role "gitlab-consul" does not exist
---- End output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ----
Ran /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul returned 2
-
Add this error during gitlab-rake geo:db:refresh_foreign_tables
.- If you are upgrading or downgrading PG version with standalone DB nodes, you may need to
gitlab-ctl restart
the read-replica DB node
- If you are upgrading or downgrading PG version with standalone DB nodes, you may need to
$ sudo gitlab-rake geo:db:refresh_foreign_tables
Refreshing foreign tables for FDW: gitlab_secondary ... rake aborted!
ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR: could not connect to server "gitlab_secondary"
DETAIL: SSL error: certificate verify failed
FATAL: no pg_hba.conf entry for host "10.138.0.59", user "gitlab", database "gitlabhq_production", SSL off
Modify https://docs.gitlab.com/ee/administration/geo/replication/updating_the_geo_nodes.html:
-
Describe and link to https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance to prevent customers from unexpectedly upgrading PG and adding a lot of downtime
Related issues
Closes #213921 (closed)
Author's checklist (required)
-
Follow the Documentation Guidelines and Style Guide. - If you have
developer
access or higher (for example, GitLab team members or Core Team members)-
Apply the documentation label, plus: - The corresponding DevOps stage and group label, if applicable.
-
development guidelines when changing docs under
doc/development/*
,CONTRIBUTING.md
, orREADME.md
. -
development guidelines and Documentation guidelines when changing docs under
development/documentation/*
. - development guidelines and Description templates (.gitlab/*) when creating/updating issue and MR description templates.
-
Assign the designated Technical Writer.
-
Do not add the feature, frontend, backend, ~"bug", or database labels if you are only updating documentation. These labels will cause the MR to be added to code verification QA issues.
When applicable:
- [-] Update the permissions table.
- [-] Link docs to and from the higher-level index page, plus other related docs where helpful.
-
Add GitLab's version history note(s). - [-] Add the product tier badge.
- [-] Add/update the feature flag section.
- [-] If you're changing document headings, search
doc/*
,app/views/*
, andee/app/views/*
for old headings replacing with the new ones to avoid broken anchors.
Review checklist
All reviewers can help ensure accuracy, clarity, completeness, and adherence to the Documentation Guidelines and Style Guide.
1. Primary Reviewer
-
Review by a code reviewer or other selected colleague to confirm accuracy, clarity, and completeness. This can be skipped for minor fixes without substantive content changes.
2. Technical Writer
-
Technical writer review. If not requested for this MR, must be scheduled post-merge. To request for this MR, assign the writer listed for the applicable DevOps stage. -
Ensure Technical Writing, documentation, and a docs::
scoped label are added. -
Add docs-only when the only files changed are under doc/*
. -
Add twdoing when starting work on the MR. -
Add twfinished if Technical Writing team work on the MR is complete but it remains open.
-
3. Maintainer
-
Review by assigned maintainer, who can always request/require the above reviews. Maintainer's review can occur before or after a technical writer review. -
Ensure a release milestone is set. -
If there has not been a technical writer review, create an issue for one using the Doc Review template.