Geo: Failure when running migrate gitlab-geo tracking database
We are running into a new issue when trying to setup Geo, currently when running rails_migration[gitlab-geo tracking]
we are hitting an error where migrations can not connect to the DB.
Thanks to @ibaum, it looks like https://gitlab.com/gitlab-org/gitlab/-/blob/a3e305b795c121fbf6b370dd7b7d18daab396358/lib/tasks/gitlab/db.rake#L143 is being called during the tracking database migrations, and it expects to connect to the main db. This goes against the current expectation and is causing the below errors.
Stack Trace
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-geo tracking] (gitlab-ee::geo_database_migrations line 25) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-geo tracking database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20220228-9579-gpbefr" ----
...
PG::ConnectionBad: could not connect to server: Connection refused
[0m Is the server running on host "0.0.0.0" and accepting
[0m TCP/IP connections on port 5432?
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `block in read_write'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:95:in `method_missing'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:59:in `with_model_connection'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:27:in `block in each_model_connection'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:21:in `each_model_connection'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning.rb:32:in `sync_partitions'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:155:in `block (3 levels) in <top (required)>'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:167:in `block (4 levels) in <top (required)>'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:93:in `block (3 levels) in <top (required)>'
[0m /opt/gitlab/embedded/bin/bundle:23:in `load'
[0m /opt/gitlab/embedded/bin/bundle:23:in `<main>'
==> /var/log/gitlab/geo-logcursor/current <==
2022-02-28_10:13:54.00043 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in `rescue in new_client': could not connect to server: Connection refused (ActiveRecord::ConnectionNotEstablished)
2022-02-28_10:13:54.00047 Is the server running on host "0.0.0.0" and accepting
2022-02-28_10:13:54.00048 TCP/IP connections on port 5432?
2022-02-28_10:13:54.00048 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'