Skip to content

Cleanup backup staging files on failure

James Fargher requested to merge backup_cleanup into master

What does this MR do and why?

Previously if any part of the backup process raised an exception, including when users cancelled a prompt, then all the backup temporary files were left behind. Here we try to ensure that these files are always cleaned up.

How to set up and validate locally

Since it's difficult to force a failure, the easiest way to check this is by aborting on the DB erase warning on restore.

  1. Make sure the GDK backup directory is empty.

    $ rm -rf tmp/backups/*

  2. Create a normal backup.

    $ bundle exec rake gitlab:backup:create
...
2022-05-17 10:46:40 +1200 -- Backup 1652741183_2022_05_17_15.0.0-pre is done.

  3. Restore from the just created backup, say no when prompted if you want to continue. Notice that there are cleanup logs after cancelling.

    $ bundle exec rake gitlab:backup:restore BACKUP=1652741183_2022_05_17_15.0.0-pre
2022-05-17 10:51:35 +1200 -- Unpacking backup ... 
2022-05-17 10:51:36 +1200 -- Unpacking backup ... done
2022-05-17 10:51:36 +1200 -- Restoring database ... 
2022-05-17 10:51:36 +1200 -- Be sure to stop Puma, Sidekiq, and any other process that
connects to the database before proceeding. For Omnibus
installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations

Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? no
2022-05-17 10:51:39 +1200 -- Quitting...
2022-05-17 10:51:39 +1200 -- Deleting tar staging files ... 
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/backup_information.yml
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/db
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/repositories
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/uploads.tar.gz
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/builds.tar.gz
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/artifacts.tar.gz
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/pages.tar.gz
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/lfs.tar.gz
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/terraform_state.tar.gz
2022-05-17 10:51:39 +1200 -- Cleaning up /home/james/src/gitlab-org/gdk/gitlab/tmp/backups/packages.tar.gz
2022-05-17 10:51:39 +1200 -- Deleting tar staging files ... done
2022-05-17 10:51:39 +1200 -- Deleting backups/tmp ... 
2022-05-17 10:51:39 +1200 -- Deleting backups/tmp ... done

  4. Check that no backup artifacts are left in the backup directory.

    Bad:

    $ ls tmp/backups/
1652741183_2022_05_17_15.0.0-pre_gitlab_backup.tar  builds.tar.gz  packages.tar.gz  terraform_state.tar.gz
artifacts.tar.gz                                    db             pages.tar.gz     uploads.tar.gz
backup_information.yml                              lfs.tar.gz     repositories

    Good:

    $ ls tmp/backups/
1652741183_2022_05_17_15.0.0-pre_gitlab_backup.tar

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Fargher

Merge request reports