pg-upgrade fails when Gitlab's PostgreSQL instance uses a non-default port

Summary

After the Gitlab package upgrade I ran pg-upgrade to bring PostgreSQL up to version 12. The post upgrade vacuum procedure fails; the process assumes PostgreSQL uses port 5432.

Steps to reproduce

To reproduce, simply change the port from the default and run the pg-upgrade procedure.

What is the current bug behavior?

The pg-upgrade fails to complete.

What is the expected correct behavior?

A successful, error-free migration.

Relevant logs

The output to stdout/stderr:

Chef Infra Client finished, 2/644 resources updated in 59 seconds Running reconfigure: OK Waiting for Database to be running. Database upgrade is complete, running vacuumdb analyze Error running /opt/gitlab/embedded/postgresql/12/bin/vacuumdb -j2 --all --analyze-in-stages -h /var/opt/gitlab/postgresql STDOUT: STDERR: vacuumdb: error: could not connect to database template1: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? Please check the output, and rerun the command as root or with sudo if needed: sudo su - gitlab-psql -c "/opt/gitlab/embedded/postgresql/12/bin/vacuumdb -j2 --all --analyze-in-stages -h /var/opt/gitlab/postgresql"

Details of package version

Provide the package version installation details

Gitlab installation:

gitlab-ce-13.4.0-ce.0.el8.x86_64 gitlab-runner-13.4.0-1.x86_64

Environment details

Operating System:
- Centos-8 (CentOS Linux release 8.2.2004 (Core) )
Installation Target, remove incorrect values:
- VM: One is a Hyper-V guest...
- The other is a KVM VPS guest.
Installation Type, remove incorrect values:
- Upgrade from version gitlab-ce-13.3.6-ce.0.el8.x86_64
Is there any other software running on the machine:
- Apache HTTPD-2.5, Apache Traffic Server 10, Tomcat, Proftpd, OpenLDAP, OpenDKIM, OpenDMARC, Postfix, Dovecot, Fail2ban, PostgreSQL, PGBouncer, Munin, PHPBB, Drupal + the usual OS stuffs and numerous custom built things
Is this a single or multiple node installation?
- Single
Resources
- CPU: Both are virtualized Pentium class CPUs
- Memory total: The KVM VPS has 8G, the DEV system has 16G

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

external_url 'https://xxxxxxxx.xxxxx.xxx'
gitlab_rails['object_store']['enabled'] = false
gitlab_rails['object_store']['connection'] = {}
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = nil
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = nil
gitlab_rails['object_store']['objects']['lfs']['bucket'] = nil
gitlab_rails['object_store']['objects']['uploads']['bucket'] = nil
gitlab_rails['object_store']['objects']['packages']['bucket'] = nil
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = nil
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = nil
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_providers'] = [
   {
     "name" => "gitlab",
     "app_id" => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
     "app_secret" => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
     "args" => { "scope" => "api"}
   }
]
gitlab_rails['db_port'] = 6432
gitlab_workhorse['enable'] = true
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "localhost:8282"
unicorn['enable'] = false
puma['enable'] = true
puma['ha'] = false
puma['worker_timeout'] = 60
puma['worker_processes'] = 2
puma['min_threads'] = 2
puma['max_threads'] = 16
puma['logrotate_postrotate'] = "gitlab-ctl hup puma"
puma['log_directory'] = "/var/log/gitlab/puma"
puma['exporter_enabled'] = true
puma['exporter_address'] = "127.0.0.1"
puma['exporter_port'] = 8083
postgresql['port'] = 6432
postgresql['data_dir'] = "/var/opt/gitlab/postgresql/data"
postgresql['shared_buffers'] = "256MB"
web_server['external_users'] = ['apache']
nginx['enable'] = false
logging['svlogd_size'] = 5 * 1024 * 1024 # rotate after 200 MB of log data
logging['svlogd_num'] = 10 # keep 30 rotated log files
logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours
logging['svlogd_filter'] = "gzip" # compress logs with gzip
logging['logrotate_frequency'] = "daily" # rotate logs daily
logging['logrotate_maxsize'] = 6 * 1024 * 1024 # rotate logs when they grow bigger than size bytes even before the specified time interval (daily, weekly, monthly, or yearly)
logging['logrotate_size'] = 5 * 1024 * 1024 # do not rotate by size by default
logging['logrotate_rotate'] = 10 # keep 30 rotated logs
logging['logrotate_compress'] = "compress" # see 'man logrotate'
logging['logrotate_method'] = "copytruncate" # see 'man logrotate'
logrotate['enable'] = true
logrotate['log_directory'] = "/var/log/gitlab/logrotate"
omnibus_gitconfig['system'] = {
    "user" => ["name = GitLab CE", "email = xxxxxx@xxxxxxxxx.xxx"],
    "receive" => ["fsckObjects = true", "advertisePushOptions = true"],
    "repack" => ["writeBitmaps = true", "threads = 8"],
    "transfer" => ["hideRefs=^refs/tmp/", "hideRefs=^refs/keep-around/", "hideRefs=^refs/remotes/"],
    "core" => ["whitespace = -blank-at-eol, -space-before-tab, -indent-with-non-tab, -tab-in-indent, -blank-at-eof, -trailing-space, -cr-at-eol, tabwidth=4"],
    "apply" => ["ignoreWhitespace = change", "whitespace = nowarn"],
    "pack" => ["threads = 8", "whitespace = nowarn"],
    "gc" => ["auto = 0", "aggressive", "prune = 2 weeks ago"],
}
mattermost['enable'] = false
prometheus['enable'] = false
prometheus['scrape_configs'] = [
    {
        'job_name': 'puma',
        'metrics_path': "/-/puma",
        'params' => {
            'param_a' => ['puma_workers'],          #Gauge  12.0    Total number of workers
            'param_b' => ['puma_running_workers'],      #Gauge  12.0    Number of booted workers
            'param_c' => ['puma_stale_workers'],        #Gauge  12.0    Number of old workers
            'param_d' => ['puma_running'],          #Gauge  12.0    Number of running threads
            'param_e' => ['puma_queued_connections'],       #Gauge  12.0    Number of connections in that worker’s “todo” set waiting for a worker thread
            'param_f' => ['puma_active_connections'],       #Gauge  12.0    Number of threads processing a request
            'param_g' => ['puma_pool_capacity'],        #Gauge  12.0    Number of requests the worker is capable of taking right now
            'param_h' => ['puma_max_threads'],      #Gauge  12.0    Maximum number of worker threads
            'param_i' => ['puma_idle_threads'],     #Gauge  12.0    Number of spawned threads which are not processing a request
            'param_j' => ['puma_killer_terminations_total'] #Gauge  12.0    Number of workers terminated by PumaWorkerKiller
        },
        'static_configs' => [
            'targets' => ['localhost:8083'],
        ],
    },
]
alertmanager['enable'] = false
grafana['enable'] = true
grafana['log_directory'] = '/var/log/gitlab/grafana'
grafana['home'] = '/var/opt/gitlab/grafana'
grafana['admin_password'] = 'xxxxxxxxx'
grafana['allow_user_sign_up'] = true
grafana['basic_auth_enabled'] = true
grafana['disable_login_form'] = false
grafana['gitlab_auth_sign_up'] = true
grafana['metrics_enabled'] = false
praefect['database_port'] = 6432
letsencrypt['enable'] = false
repmgr['port'] = 6432

Edited Sep 23, 2020 by Randy DuCharme