Upgrade from 9.5.5 CE to 10 CE fails on CentOS 7 with non-bundled PostgreSQL
Summary
yum update -y gitlab-ce to upgrade from an omnibus ce installation on my test machine 9.5.5 to 10.0.0 fails. I'm using Postgres yum repo for latest postgresql-server and gitlab.rb is configured to use the non-bundled database server which worked fine in every past release upgrade.
CentOS version information:
# cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)Yum log:
# yum update -y gitlab-ce grafana
...
Total download size: 393 M
Is this ok [y/d/N]: y
Downloading packages:
...
gitlab preinstall:
gitlab preinstall: This node does not appear to be running a database
gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab preinstall:
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
[SKIPPED]
Dumping uploads ...
[SKIPPED]
Dumping builds ...
[SKIPPED]
Dumping artifacts ...
[SKIPPED]
Dumping pages ...
[SKIPPED]
Dumping lfs objects ...
[SKIPPED]
Dumping container registry images ...
[SKIPPED]
Creating backup archive: 1506167770_2017_09_23_9.5.5_gitlab_backup.tar ... done
Uploading backup archive to remote storage ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... done. (1 removed)
Updating : gitlab-ce-10.0.0-ce.0.el7.x86_64 10/16
Updating : grafana-4.5.2-1.x86_64 11/16
Cleanup : gitlab-ce-9.5.5-ce.0.el7.x86_64 12/16
Cleanup : grafana-4.5.1-1.x86_64 13/16
Cleanup : policycoreutils-2.5-11.el7_3.x86_64 14/16
Cleanup : libsemanage-2.5-5.1.el7_3.x86_64 15/16
Cleanup : audit-libs-2.6.5-3.el7_3.1.x86_64 16/16
Checking PostgreSQL executables: OK
Shutting down all GitLab services except those needed for migrations
ok: down: gitaly: 0s, normally up
ok: down: gitlab-monitor: 0s, normally up
ok: down: gitlab-pages: 0s, normally up
ok: down: gitlab-workhorse: 0s, normally up
ok: down: logrotate: 1s, normally up
timeout: run: mailroom: (pid 10735) 79173s, want down, got TERM
ok: down: mattermost: 1s, normally up
ok: down: nginx: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: registry: 0s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: unicorn: 1s, normally up
Reconfiguring GitLab to apply migrations
/usr/share/rubygems/rubygems/dependency.rb:296:in `to_specs': Could not find 'chef' (>= 0.a) among 5 total gem(s) (Gem::LoadError)
from /usr/share/rubygems/rubygems/dependency.rb:307:in `to_spec'
from /usr/share/rubygems/rubygems/core_ext/kernel_gem.rb:47:in `gem'
from /opt/gitlab/embedded/bin/chef-client:25:in `<main>'
Ensuring PostgreSQL is updated:/usr/share/rubygems/rubygems/dependency.rb:296:in `to_specs': Could not find 'omnibus-ctl' (>= 0.a) among 5 total gem(s) (Gem::LoadError)
from /usr/share/rubygems/rubygems/dependency.rb:307:in `to_spec'
from /usr/share/rubygems/rubygems/core_ext/kernel_gem.rb:47:in `gem'
from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
warning: %posttrans(gitlab-ce-10.0.0-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-10.0.0-ce.0.el7.x86_64
POSTTRANS: Running script
Verifying : audit-libs-2.7.6-3.el7.x86_64 1/16
Verifying : libsemanage-python-2.5-8.el7.x86_64 2/16
Verifying : audit-libs-python-2.7.6-3.el7.x86_64 3/16
Verifying : libsemanage-2.5-8.el7.x86_64 4/16
Verifying : grafana-4.5.2-1.x86_64 5/16
Verifying : python-IPy-0.75-6.el7.noarch 6/16
Verifying : policycoreutils-python-2.5-17.1.el7.x86_64 7/16
Verifying : gitlab-ce-10.0.0-ce.0.el7.x86_64 8/16
Verifying : libcgroup-0.41-13.el7.x86_64 9/16
Verifying : setools-libs-3.3.8-1.1.el7.x86_64 10/16
Verifying : policycoreutils-2.5-17.1.el7.x86_64 11/16
Verifying : policycoreutils-2.5-11.el7_3.x86_64 12/16
Verifying : gitlab-ce-9.5.5-ce.0.el7.x86_64 13/16
Verifying : audit-libs-2.6.5-3.el7_3.1.x86_64 14/16
Verifying : libsemanage-2.5-5.1.el7_3.x86_64 15/16
Verifying : grafana-4.5.1-1.x86_64Output of checks
After yum command failed, i'm no longer able to run gitlab-ctl or gitlab-rake. See:
gitlab-ctl
/usr/share/rubygems/rubygems/dependency.rb:296:in `to_specs': Could not find 'omnibus-ctl' (>= 0.a) among 5 total gem(s) (Gem::LoadError)
from /usr/share/rubygems/rubygems/dependency.rb:307:in `to_spec'
from /usr/share/rubygems/rubygems/core_ext/kernel_gem.rb:47:in `gem'
from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'Possible fixes
Downgrade does not work unless you skip the auto-migration:
# sudo touch /etc/gitlab/skip-auto-migrations
# yum downgrade gitlab-ce
...
Resolving Dependencies
--> Running transaction check
---> Package gitlab-ce.x86_64 0:9.5.5-ce.0.el7 will be a downgrade
---> Package gitlab-ce.x86_64 0:10.0.0-ce.0.el7 will be erased
--> Finished Dependency Resolution
...
Total download size: 381 M
Is this ok [y/d/N]: y
Downloading packages:
gitlab-ce-9.5.5-ce.0.el7.x86_64.rpm | 381 MB 00:00:37
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : gitlab-ce-9.5.5-ce.0.el7.x86_64 1/2
*. *.
*** ***
***** *****
.****** *******
******** ********
,,,,,,,,,***********,,,,,,,,,
,,,,,,,,,,,*********,,,,,,,,,,,
.,,,,,,,,,,,*******,,,,,,,,,,,,
,,,,,,,,,*****,,,,,,,,,.
,,,,,,,****,,,,,,
.,,,***,,,,
,*,.
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \
/ /_/ / / /_/ /___/ /_/ / /_/ /
\____/_/\__/_____/\__,_/_.___/
gitlab: Thank you for installing GitLab!
gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:
sudo gitlab-ctl reconfigure
gitlab: GitLab should be reachable at https://git.dev.zgrp.net
gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab: And running reconfigure again.
gitlab:
gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab:
gitlab: WARNING:
gitlab: GitLab now ships with a newer version of PostgreSQL (9.6.3) by default
gitlab: We did not upgrade since /etc/gitlab/skip-automigrations exists
gitlab: To manuall upgrade, RUN THE FOLLOWING COMMAND:
sudo gitlab-ctl pg-upgrade
gitlab: In the future, we will be removing old versions of PostgreSQL from the omnibus package
gitlab: Hosts which have not upgraded to the new version will begin to fail then
gitlab: For more details, please see:
gitlab: https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server
gitlab:
Cleanup : gitlab-ce-10.0.0-ce.0.el7.x86_64 2/2
Checking PostgreSQL executables: OK
Found /etc/gitlab/skip-auto-migrations, exiting...
Verifying : gitlab-ce-9.5.5-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-10.0.0-ce.0.el7.x86_64 2/2
Removed:
gitlab-ce.x86_64 0:10.0.0-ce.0.el7
Installed:
gitlab-ce.x86_64 0:9.5.5-ce.0.el7
Complete!Since no database migraten ever run to upgrade to 10.0.0 this solved my issues completely, but for now I'm stuck on the 9.5.x version. Any hints how to upgrade would be really appreciated.