Enable incremental repository backups

What does this MR do and why?

#355945 (closed)

Enabling this feature changes the layout for all repository backups (full or incremental) from the legacy layout to the pointer layout.

This feature is not used on .com so can only be rolled out by turning the feature on by default, the feature is provided to give on-prem users a way to revert the behaviour in case something goes wrong.

How to set up and validate locally

There are two main scenarios here. First checking the feature itself, then checking the upgrade path.

Before each of these scenarios it might be a good idea to clear out the backup directory so the tests do not get cross contaminated. From your gitlab directory:

$ rm -rf tmp/backup/*

Create incremental backups

1. Remove the flag if already set.

   Feature.remove(:incremental_repository_backup)

2. Create a full backup.

   $ bundle exec rake gitlab:backup:create
   ...
   2022-04-07 10:57:50 +1200 -- Backup 1649285835_2022_04_07_14.10.0-pre is done.

3. Find a target project to inspect.

   [1] pry(main)> Project.find(20).repository.relative_path
   => "@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git"

4. Push changes to the target repo (anything except a branch delete).
5. Take an incremental backup. Note, the backup timestamp printed at the end is the same as the backup gets overwritten.

   $ bundle exec rake gitlab:backup:create INCREMENTAL=yes BACKUP=1649285835_2022_04_07_14.10.0-pre
   ...
   2022-04-07 11:01:15 +1200 -- Backup 1649285835_2022_04_07_14.10.0-pre is done.

6. Check that a new increment was added for the target repo. We expect to see files named 002. Strip off the .git from the target repo above and add it to repositories/ for the directory within the tar file.

   $ tar tvf tmp/backups/1649285835_2022_04_07_14.10.0-pre_gitlab_backup.tar repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b
   drwx------ james/james       0 2022-04-07 10:57 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/
   drwx------ james/james       0 2022-04-07 11:01 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/1649285835_2022_04_07_14.10.0-pre/
   -rw------- james/james  824684 2022-04-07 11:01 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/1649285835_2022_04_07_14.10.0-pre/002.refs
   -rw------- james/james  824684 2022-04-07 10:57 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/1649285835_2022_04_07_14.10.0-pre/001.refs
   -rw------- james/james 1319747386 2022-04-07 10:57 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/1649285835_2022_04_07_14.10.0-pre/001.bundle
   -rw------- james/james          3 2022-04-07 10:57 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/1649285835_2022_04_07_14.10.0-pre/LATEST
   -rw------- james/james         17 2022-04-07 11:01 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/1649285835_2022_04_07_14.10.0-pre/002.bundle
   -rw------- james/james         33 2022-04-07 10:57 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b/LATEST

Test that we can restore a legacy backup

1. Make sure the flag is disabled.

   Feature.disable(:incremental_repository_backup)

2. Create a legacy backup

   $ bundle exec rake gitlab:backup:create
   ...
   2022-04-07 11:11:20 +1200 -- Backup 1649286644_2022_04_07_14.10.0-pre is done.

3. Check that it was indeed a legacy backup. There should not be any files named 001.bundle. You can use the same tar command as per the previous scenario, you just need to remove the last path segment from the repository path.

   $ tar tvf tmp/backups/1649286644_2022_04_07_14.10.0-pre_gitlab_backup.tar repositories/@hashed/f5/ca/
   drwx------ james/james       0 2022-04-07 11:10 repositories/@hashed/f5/ca/
   -rw------- james/james 1319747386 2022-04-07 11:11 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.bundle
   -rw------- james/james     824684 2022-04-07 11:10 repositories/@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.refs

4. Remove the flag so that it is enabled by default.

   Feature.remove(:incremental_repository_backup)

5. Restore from the legacy backup.

   $ bundle exec rake gitlab:backup:restore BACKUP=1649286644_2022_04_07_14.10.0-pre
   ...
   2022-04-07 11:18:27 +1200 -- Restore task is done.

6. Make sure your repositories in GDK look normal.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.