Broken migration due to not-null constraint
Summary
Updating from GitLab CE v10.7.1 to GitLab CE v10.8.1 using the Omnibus package results in a database migration failure.
Steps to reproduce
Upgrade GitLab CE to v10.8.1 and run sudo -H gitlab-ctl reconfigue
(though dpkg
will typically run this automatically). There are presumably some specifics of our installation which trigger the issue for us but not others. Our GitLab installation has gone through many updates dating back years.
What is the current bug behavior?
A database migration failure occurs during upgrade while applying 20180413022611_create_missing_namespace_for_internal_users.rb
. The issue is caused by an attempted insertion of a row to the namespaces
table without setting the created_at
and updated_at
fields. This results in a not-null constraint violation.
What is the expected correct behavior?
Migration runs successfully.
Relevant logs and/or screenshots
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20180528-60716-o28qkq" ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::NotNullViolation: ERROR: null value in column "created_at" violates not-null constraint
DETAIL: Failing row contains (54, ghost, ghost, 52, null, null, null, , null, f, 20, f, null, null, null, f, 48, null, null).
: INSERT INTO namespaces(owner_id, path, name, created_at) VALUES(52, 'ghost', 'ghost', NOW()) RETURNING "id"
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:49:in `create_namespace'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:6:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR: null value in column "created_at" violates not-null constraint
DETAIL: Failing row contains (54, ghost, ghost, 52, null, null, null, , null, f, 20, f, null, null, null, f, 48, null, null).
: INSERT INTO namespaces(owner_id, path, name, created_at) VALUES(52, 'ghost', 'ghost', NOW()) RETURNING "id"
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:49:in `create_namespace'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:6:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::NotNullViolation: ERROR: null value in column "created_at" violates not-null constraint
DETAIL: Failing row contains (54, ghost, ghost, 52, null, null, null, , null, f, 20, f, null, null, null, f, 48, null, null).
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:49:in `create_namespace'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:6:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20180413022611 CreateMissingNamespaceForInternalUsers: migrating ===========
-- column_exists?(:users, :support_bot)
-> 0.0021s
STDERR:
---- End output of "bash" "/tmp/chef-script20180528-60716-o28qkq" ----
Ran "bash" "/tmp/chef-script20180528-60716-o28qkq" returned 1
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.3.7p456 Gem Version: 2.6.14 Bundler Version:1.13.7 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.16.3 Sidekiq Version:5.0.5 Go Version: unknownGitLab information Version: 10.8.1 Revision: 21a8d61 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.corp..com HTTP Clone URL: https://gitlab.corp..com/some-group/some-project.git SSH Clone URL: git@git.corp..com:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 3/3 ... ok 5/6 ... ok 5/7 ... ok 5/8 ... ok 5/9 ... ok 5/11 ... ok 5/13 ... ok 5/14 ... ok 7/15 ... ok 7/16 ... ok 8/17 ... ok 8/18 ... ok 17/19 ... ok 9/20 ... ok 9/21 ... ok 5/22 ... ok 5/23 ... ok 5/24 ... ok 17/25 ... ok 5/26 ... ok 17/27 ... ok 4/28 ... ok 8/29 ... ok 9/30 ... ok 5/31 ... ok 17/32 ... ok 5/34 ... ok 5/35 ... ok 16/36 ... ok 3/37 ... ok 17/38 ... ok 12/39 ... ok 5/43 ... ok 5/45 ... ok 3/46 ... ok 5/49 ... ok 5/50 ... ok 27/51 ... ok 17/52 ... ok 17/53 ... ok 5/55 ... ok 6/56 ... ok 6/57 ... ok 5/59 ... ok 12/60 ... ok 3/61 ... ok 4/62 ... ok 5/63 ... ok 5/64 ... ok 5/65 ... ok 5/66 ... ok 6/67 ... ok 3/68 ... ok 9/69 ... ok 5/70 ... ok 6/71 ... ok 29/72 ... ok 29/73 ... ok 11/74 ... repository is empty 3/75 ... ok 8/77 ... ok 5/79 ... ok 6/80 ... ok 4/81 ... ok 7/82 ... ok 5/83 ... ok 5/84 ... ok 8/85 ... ok 5/87 ... ok 12/88 ... ok 12/90 ... ok 27/91 ... ok 32/92 ... repository is empty 12/97 ... ok 17/98 ... ok 5/99 ... ok 3/100 ... ok 17/101 ... ok 12/103 ... ok 17/104 ... ok 12/106 ... ok 8/108 ... ok 27/109 ... ok 27/110 ... ok 8/111 ... ok 17/112 ... ok 8/113 ... ok 8/114 ... ok 8/115 ... ok 12/116 ... ok 12/117 ... ok 17/118 ... ok 27/119 ... ok 17/120 ... ok 8/121 ... ok 12/122 ... ok 17/124 ... ok 17/125 ... ok 5/126 ... repository is empty 5/127 ... ok 37/128 ... ok 12/129 ... ok 12/130 ... ok 7/131 ... ok 17/132 ... ok 17/133 ... ok 17/134 ... ok 12/135 ... ok 5/137 ... ok 45/138 ... ok 17/139 ... ok 8/141 ... ok 8/142 ... ok 27/143 ... ok 2/144 ... ok 8/146 ... ok 17/147 ... ok 17/148 ... ok 47/149 ... ok 12/150 ... ok 47/151 ... ok 48/152 ... ok 48/153 ... ok 48/154 ... ok 48/155 ... ok 5/157 ... ok 8/158 ... ok 8/159 ... ok 5/160 ... ok 8/161 ... ok 45/162 ... ok 50/163 ... ok 2/164 ... ok 2/165 ... ok 5/166 ... ok 50/167 ... ok 50/168 ... ok 8/169 ... ok 50/170 ... ok 5/171 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
Server: ldapmain not verifying SSL hostname of LDAPS server '.com:389' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results)
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/3 ... yes 5/6 ... yes 5/7 ... yes 5/8 ... yes 5/9 ... yes 5/11 ... yes 5/13 ... yes 5/14 ... yes 7/15 ... yes 7/16 ... yes 8/17 ... yes 8/18 ... yes 17/19 ... yes 9/20 ... yes 9/21 ... yes 5/22 ... yes 5/23 ... yes 5/24 ... yes 17/25 ... yes 5/26 ... yes 17/27 ... yes 4/28 ... yes 8/29 ... yes 9/30 ... yes 5/31 ... yes 17/32 ... yes 5/34 ... yes 5/35 ... yes 16/36 ... yes 3/37 ... yes 17/38 ... yes 12/39 ... yes 5/43 ... yes 5/45 ... yes 3/46 ... yes 5/49 ... yes 5/50 ... yes 27/51 ... yes 17/52 ... yes 17/53 ... yes 5/55 ... yes 6/56 ... yes 6/57 ... yes 5/59 ... yes 12/60 ... yes 3/61 ... yes 4/62 ... yes 5/63 ... yes 5/64 ... yes 5/65 ... yes 5/66 ... yes 6/67 ... yes 3/68 ... yes 9/69 ... yes 5/70 ... yes 6/71 ... yes 29/72 ... yes 29/73 ... yes 11/74 ... yes 3/75 ... yes 8/77 ... yes 5/79 ... yes 6/80 ... yes 4/81 ... yes 7/82 ... yes 5/83 ... yes 5/84 ... yes 8/85 ... yes 5/87 ... yes 12/88 ... yes 12/90 ... yes 27/91 ... yes 32/92 ... yes 12/97 ... yes 17/98 ... yes 5/99 ... yes 3/100 ... yes 17/101 ... yes 12/103 ... yes 17/104 ... yes 12/106 ... yes 8/108 ... yes 27/109 ... yes 27/110 ... yes 8/111 ... yes 17/112 ... yes 8/113 ... yes 8/114 ... yes 8/115 ... yes 12/116 ... yes 12/117 ... yes 17/118 ... yes 27/119 ... yes 17/120 ... yes 8/121 ... yes 12/122 ... yes 17/124 ... yes 17/125 ... yes 5/126 ... yes 5/127 ... yes 37/128 ... yes 12/129 ... yes 12/130 ... yes 7/131 ... yes 17/132 ... yes 17/133 ... yes 17/134 ... yes 12/135 ... yes 5/137 ... yes 45/138 ... yes 17/139 ... yes 8/141 ... yes 8/142 ... yes 27/143 ... yes 2/144 ... yes 8/146 ... yes 17/147 ... yes 17/148 ... yes 47/149 ... yes 12/150 ... yes 47/151 ... yes 48/152 ... yes 48/153 ... yes 48/154 ... yes 48/155 ... yes 5/157 ... yes 8/158 ... yes 8/159 ... yes 5/160 ... yes 8/161 ... yes 45/162 ... yes 50/163 ... yes 2/164 ... yes 2/165 ... yes 5/166 ... yes 50/167 ... yes 50/168 ... yes 8/169 ... yes 50/170 ... yes 5/171 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.7) Git version >= 2.9.5 ? ... yes (2.16.3) Git user has default SSH configuration? ... yes Active users: ... 19
Checking GitLab ... Finished
Possible fixes
The problem line in the migration appears to be here. We were able to workaround the issue by modifying it as follows:
48c48
< insert_query = "INSERT INTO namespaces(owner_id, path, name, created_at, updated_at) VALUES(#{user_id}, '#{path}', '#{path}', NOW(), NOW())"
---
> insert_query = "INSERT INTO namespaces(owner_id, path, name) VALUES(#{user_id}, '#{path}', '#{path}')"