Skip to content

Create feature flag for incremental repository backups

James Fargher requested to merge incremental_repository_backup into master

What does this MR do and why?

#351383 (closed)

  • Adds a feature flag named incremental_repository_backup.
  • When enabled this will provide the minimum required arguments to gitaly-backup to enable incremental repository backups.
  • The approach may change as we learn how exactly the incremental backup workflow will work.

How to set up and validate locally

  1. Enable the feature
    Feature.enable(:incremental_repository_backup)
  2. Take a normal backup. Note that we cannot pack into a tar file since incremental backups need access to the previous increment and it doesn't know about tar files.
    $ bundle exec rake gitlab:backup:create SKIP=tar
  3. Find a target project to inspect.
    [1] pry(main)> Project.find(20).repository.relative_path
    => "@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git"
  4. Check that the pointer layout was indeed used. Strip off the .git from above and add it to tmp/backups/repositories/ We expect to a file named LATEST. These are the "pointers".
    $ tree tmp/backups/repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/
    tmp/backups/repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/
    ├── 20220316012048
    │   ├── 001.bundle
    │   ├── 001.refs
    │   └── LATEST
    └── LATEST
    
    1 directory, 4 files
  5. Push changes to the target repo (anything except a branch delete).
  6. Take an incremental backup.
    $ bundle exec rake gitlab:backup:create SKIP=tar INCREMENTAL=yes
  7. Check that a new increment was created. We expect to see files named 002.
    $ tree tmp/backups/repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/
    tmp/backups/repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/
    ├── 20220316012048
    │   ├── 001.bundle
    │   ├── 001.refs
    │   ├── 002.bundle
    │   ├── 002.refs
    │   └── LATEST
    └── LATEST
    
    1 directory, 6 files
    🎉
  8. Restore from said backup. Hopefully this wont destroy GDK 😄
    $ bundle exec rake gitlab:backup:restore
    ...
    Restore task is done.
  9. Check that the target repository looks okay.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Fargher

Merge request reports

Loading