Skip to content

Unpack a previous backup for incremental backup

James Fargher requested to merge backup_incremental into master

What does this MR do and why?

#351383 (closed)

Creating incremental backups requires access to the previous backup in order to create a differential. To allow this, previously creating an incremental backup required SKIP=tar. Instead of forcing this requirement, here we are unpacking a specified backup first.

We're assuming here that it's okay to overwrite the previous backup tar file. Most of the backup tasks do not support incremental backups yet. So these will essentially be writing a new full backup over the last full backup.

How to set up and validate locally

  1. Make sure the feature is enabled.
    Feature.enable(:incremental_repository_backup)
  2. Create a full backup.
    $ bundle exec rake gitlab:backup:create
    ...
    2022-03-24 14:53:58 +1300 -- Backup 1648086824_2022_03_24_14.10.0-pre is done.
  3. Find a target project to inspect.
    [1] pry(main)> Project.find(20).repository.relative_path
    => "@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git"
  4. Push changes to the target repo (anything except a branch delete).
  5. Take an incremental backup. Notice that the backup timestamp printed at the end is the same, the backup gets overwritten.
    $ bundle exec rake gitlab:backup:create INCREMENTAL=yes BACKUP=1648086824_2022_03_24_14.10.0-pre
    ...
    2022-03-24 14:56:47 +1300 -- Backup 1648086824_2022_03_24_14.10.0-pre is done.
  6. Check that a new increment was added for the target repo. We expect to see files named 002. Strip off the .git from the target repo above and add it to repositories/ for the directory within the tar file.
    $ tar tvf tmp/backups/1648086824_2022_03_24_14.10.0-pre_gitlab_backup.tar 
    repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b
    drwx------ james/james       0 2022-03-24 14:53 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/
    drwx------ james/james       0 2022-03-24 14:56 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/20220324015346/
    -rw------- james/james  824684 2022-03-24 14:56 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/20220324015346/002.refs
    -rw------- james/james  824684 2022-03-24 14:53 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/20220324015346/001.refs
    -rw------- james/james 1319756122 2022-03-24 14:53 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/20220324015346/001.bundle
    -rw------- james/james          3 2022-03-24 14:53 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/20220324015346/LATEST
    -rw------- james/james         17 2022-03-24 14:56 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/20220324015346/002.bundle
    -rw------- james/james         14 2022-03-24 14:53 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/LATEST

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Fargher

Merge request reports

Loading