Using Patroni casues 'cannot execute CREATE EXTENSION in a read-only transaction' when run as part of clustered environment
Summary
Whilst trying to set up a clustered environment to use Patroni we are running into the below issue whilst running a reconfigure on the primary rails node. The server architecture is setup the same as the 3k Reference Architecture with 3 Postgres servers and 3 Rails servers.
gitlab_rails['auto_migrate']
is set to false on all servers with the exception of the primary rails node where it is true.
Steps to reproduce
Create a new environment using the Performance Environment Builder based on an architecture for at least 3k users with install_patroni: true
set in your vars.yml file.
Or manually create a clustered environment with a similar architecture to that of the 3k reference environment.
What is the current bug behavior?
When running gitlab-ctl reconfigure
on the primary rails node it is failing with the error
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] psql:/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql:3: ERROR: cannot execute CREATE EXTENSION in a read-only transaction
rake aborted!
failed to execute:
psql -v ON_ERROR_STOP=1 -q -X -f /opt/gitlab/embedded/service/gitlab-rails/db/structure.sql gitlabhq_production
Please check the output above for any errors and make sure that `psql` is installed in your PATH and has proper permissions.
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:structure:load
(See full trace by running task with --trace)
What is the expected correct behavior?
The database should not be locked in read only mode and the migration should be able to run resulting in the reconfigure passing.
Relevant logs and/or screenshots
pgbouncer show servers & show databases
pgbouncer=# SHOW SERVERS;
type | user | database | state | addr | port | local_addr | local_port | connect_time |
request_time | wait | wait_us | close_needed | ptr | link | remote_pid | tls
------+-----------+---------------------+-------+-------------+------+--------------+------------+-------------------------+--
-----------------------+------+---------+--------------+----------------+------+------------+-----
S | gitlab | gitlabhq_production | idle | 10.142.0.67 | 5432 | 10.142.0.112 | 51774 | 2020-06-24 13:23:13 UTC | 2
020-06-24 13:28:23 UTC | 0 | 0 | 0 | 0x555e8cc9f060 | | 27823 |
S | pgbouncer | gitlabhq_production | idle | 10.142.0.67 | 5432 | 10.142.0.112 | 51688 | 2020-06-24 13:14:16 UTC | 2
020-06-24 13:28:23 UTC | 0 | 0 | 0 | 0x555e8cc9ec40 | | 27366 |
(2 rows)
pgbouncer=# SHOW DATABASES;
name | host | port | database | force_use
r | pool_size | reserve_pool | pool_mode | max_connections | current_connections | paused | disabled
---------------------+----------------------------------------------------------------+------+---------------------+----------
--+-----------+--------------+-----------+-----------------+---------------------+--------+----------
gitlabhq_production | omnibus-3k-postgres-2.c.gitlab-qa-distribution-35632a.internal | 5432 | gitlabhq_production |
| 100 | 5 | | 100 | 2 | 0 | 0
pgbouncer | | 6432 | pgbouncer | pgbouncer
| 2 | 0 | statement | 100 | 0 | 0 | 0
(2 rows)
Postgress members
+----------------------+----------------------------------------------------------------+-------------+--------+---------+----+-----------+
| Cluster | Member | Host | Role | State | TL | Lag in MB |
+----------------------+----------------------------------------------------------------+-------------+--------+---------+----+-----------+
| gitlab-postgresql-ha | omnibus-3k-postgres-1.c.gitlab-qa-distribution-35632a.internal | 10.142.0.65 | | running | 2 | 0 |
| gitlab-postgresql-ha | omnibus-3k-postgres-2.c.gitlab-qa-distribution-35632a.internal | 10.142.0.67 | Leader | running | 2 | |
| gitlab-postgresql-ha | omnibus-3k-postgres-3.c.gitlab-qa-distribution-35632a.internal | 10.142.0.72 | | running | 2 | 0 |
+----------------------+----------------------------------------------------------------+-------------+--------+---------+----+-----------+