Skip to content

Initial reconfigure does not restart unicorn for external geo tracking database

Summary

This looks like some sort of race condition where a Geo secondary container that uses a gitlab.rb configured with an external DB and an external tracking DB doesn't properly restart the unicorn service after creating the database_geo.yml file.

We have this dependent services logic to restart unicorn, but in reproducing this, I couldn't see any output in the reconfigure hinting that unicorn is being restarted, maybe a case of should_notify? not being ran in the task and returning false. (Being ran outside the task, so computed at the beginning of the reconfigure and being false because the services are not up then? Though, I'm not sure if that's exactly the case)

Because unicorn is not restarted, the Rails initializer won't run, and GitLab will error out further with Geo database configuration file is missing. errors because of this.

cc @dbalexandre

Steps to reproduce

  1. Have a Geo setup with external DBs (including the tracking DB)
  2. Use the secondary gitlab.rb to start up a Docker Omnibus container and notice the error

What is the current bug behavior?

Unicorn doesn't get restarted after database_geo.yml is created.

What is the expected correct behavior?

Unicorn gets restarted and the Rails app uses the database_geo.yml as expected.

Possible workaround at the moment

Manually restart unicorn after the container init.

Relevant logs

Relevant logs 
Initial reconfigure logs:



Recipe: repmgr::repmgrd_disable
  * runit_service[repmgrd] action disable
    * ruby_block[disable repmgrd] action run (skipped due to only_if)
     (up to date)
Recipe: gitlab-ee::geo-secondary
  * templatesymlink[Create a database_geo.yml and create a symlink to Rails root] action create
    * template[/var/opt/gitlab/gitlab-rails/etc/database_geo.yml] action create
      - create new file /var/opt/gitlab/gitlab-rails/etc/database_geo.yml
      - update content in file /var/opt/gitlab/gitlab-rails/etc/database_geo.yml from none to bc01d1
      --- /var/opt/gitlab/gitlab-rails/etc/database_geo.yml     2020-03-11 17:14:03.544491588 +0000
      +++ /var/opt/gitlab/gitlab-rails/etc/.chef-database_geo20200311-22-b2gvy3.yml     2020-03-11 17:14:03.544491588 +0000
      @@ -1 +1,24 @@
      +# This file is managed by gitlab-ctl. Manual changes will be
      +# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
      +# and run `sudo gitlab-ctl reconfigure`.
      +
      +production:
      +  adapter: postgresql
      +  encoding: unicode
      +  collation:
      +  database: gitlabhq_geo_production
      +  pool: 1
      +  username: "postgres"
      +  password: "****"
      +  host: "****"
      +  port: 5432
      +  socket:
      +  sslmode:
      +  sslcompression: 0
      +  sslrootcert:
      +  sslca:
      +  load_balancing: {"hosts":[]}
      +  prepared_statements:
      +  statement_limit:
      +  fdw: true
      - change mode from '' to '0640'
      - change owner from '' to 'root'
      - change group from '' to 'git'
    * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database_geo.yml to /var/opt/gitlab/gitlab-rails/etc/database_geo.yml] action create
      - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/database_geo.yml to /var/opt/gitlab/gitlab-rails/etc/database_geo.yml

  * file[/opt/gitlab/embedded/service/gitlab-rails/ee/db/geo/schema.rb] action create
    - change owner from 'root' to 'git'
Recipe: gitlab-ee::geo_database_migrations
  * bash[migrate gitlab-geo tracking database] action run
    - execute "bash"  "/tmp/chef-script20200311-22-1o7zd74"

Details of package version

Provide the package version installation details 
Docker: gitlab/gitlab-ee:12.8.5-ee.0

Environment details

  • Operating System: Ubuntu host, Docker image
  • Installation Target, remove incorrect values:
    • Other: AWS host running the docker container, RDS for external DBs
  • Installation Type, remove incorrect values:
    • New Installation
  • Is there any other software running on the machine: no
  • Is this a single or multiple node installation? Geo, secondary

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 


roles ['geo_secondary_role']

gitlab_rails['db_host'] = '****'
gitlab_rails['db_username'] = 'postgres'
gitlab_rails['db_password'] = '****'
gitlab_rails['db_port'] = '5432'

geo_secondary['db_username'] = 'postgres'
geo_secondary['db_password'] = '****'
geo_secondary['db_port'] = '5432'

geo_secondary['db_host'] = '****'
geo_postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
Edited by Catalin Irimie