A database failed backup should return non-zero exit code
Summary
Backup process exit code should reflect the presence of error. Especially as it is a "preinstall" step during an omnibus package upgrade, a failure in the backup process should lead to an interruption of the upgrade.
(Context : I had created an extra table in the Gitlab database for analytics purposes, thus the owner was gitlab-psql
instead of gitlab
(my bad). A few days after, forgetting this table, I launched an apt-get upgrade
to upgrade Gitlab, and the pg_dump
could not run, so I had no backup of before the upgrade, in case the migrations went wrong or something)
Steps to reproduce
- Do something that leads to
pg_dump
failing (for instance:sudo gitlab-psql -c 'create table temp (test VARCHAR);'
) - Trigger a backup including the database :
sudo gitlab-backup create SKIP=uploads,builds,artifacts,lfs,terraform_state,registry,pages,repositories,packages
What is the current bug behavior?
The command exits with 0
, despite the failure of pg_dump
and the emptiness and the database backup
What is the expected correct behavior?
The command exits with 1
.
Relevant logs and/or screenshots
$ sudo gitlab-backup create SKIP=uploads,builds,artifacts,lfs,terraform_state,registry,pages,repositories,packages
2023-05-15 21:22:13 +0200 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: query failed: ERROR: permission denied for table temp
pg_dump: error: query was: LOCK TABLE public.temp IN ACCESS SHARE MODE
2023-05-15 21:22:13 +0200 -- Dumping database failed: Failed to create compressed file '/var/opt/gitlab/backups/db/database.sql.gz' when trying to backup the main database:
- host: '/var/opt/gitlab/postgresql'
- port: '5432'
- database: 'gitlabhq_production'
2023-05-15 21:22:13 +0200 -- Dumping repositories ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping uploads ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping builds ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping artifacts ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping pages ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping lfs objects ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping terraform states ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping container registry images ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Dumping packages ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Creating backup archive: 1684178533_2023_05_15_15.11.3_gitlab_backup.tar ...
2023-05-15 21:22:13 +0200 -- Creating backup archive: 1684178533_2023_05_15_15.11.3_gitlab_backup.tar ... done
2023-05-15 21:22:13 +0200 -- Uploading backup archive to remote storage ... [SKIPPED]
2023-05-15 21:22:13 +0200 -- Deleting old backups ...
2023-05-15 21:22:13 +0200 -- Deleting old backups ... done. (0 removed)
2023-05-15 21:22:13 +0200 -- Deleting tar staging files ...
2023-05-15 21:22:13 +0200 -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2023-05-15 21:22:13 +0200 -- Cleaning up /var/opt/gitlab/backups/db
2023-05-15 21:22:13 +0200 -- Deleting tar staging files ... done
2023-05-15 21:22:13 +0200 -- Deleting backups/tmp ...
2023-05-15 21:22:13 +0200 -- Deleting backups/tmp ... done
2023-05-15 21:22:13 +0200 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2023-05-15 21:22:13 +0200 -- Backup 1684178533_2023_05_15_15.11.3 is done.
2023-05-15 19:22:13 +0000 -- Deleting backup and restore lock file
$ echo $?
0
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 10 Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.2.33 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.11.3 Revision: 332cca52b7e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://MASKED HTTP Clone URL: https://MASKED/some-group/some-project.git SSH Clone URL: git@MASKED:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 14.18.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Possible fixes
Reraise Backup::DatabaseBackupError
at https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/backup/manager.rb#L73