Write ref list when taking backups
This is the beginning of the infrastructure we need to do incremental backups. Write refs so that we can calculate patterns to give to git bundle create
to only backup new objects and switching to use CreateBundleFromRefList
instead of assuming we'll backup all the objects all the time.
We want to start generating ref files now, so that if/when we start doing incremental backups most installations will already have refs available.
Ideally we would stop overwriting the old bundles/refs too but we can't until the old ruby implementation is deleted. Plus once we start restoring incremental backups we will need to be able to get a listing from the sink which hasn't been implemented yet.
Some small testing with the gitlab repo shows no significant change in performance.
Before:
$ time ./_build/bin/gitaly-backup create -path=/home/james/Downloads/backup < ~/Downloads/single.json
INFO[0000] started backup gl_project_path=root/gitlab relative_path=@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git storage_name=default
INFO[0031] completed backup gl_project_path=root/gitlab relative_path=@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git storage_name=default
real 0m31.965s
user 0m7.051s
sys 0m4.912s
After:
$ time ./_build/bin/gitaly-backup create -path=/home/james/Downloads/backup < ~/Downloads/single.json
INFO[0000] started backup gl_project_path=root/gitlab relative_path=@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git storage_name=default
INFO[0030] completed backup gl_project_path=root/gitlab relative_path=@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git storage_name=default
real 0m30.287s
user 0m7.576s
sys 0m4.240s