programmatically testing backup and restore

In order to avoid relying on user feedback as a mean of improving reliability of the backup and restore

We need to have an appropriate method of testing that functionality.

The scope of this issue is investigating possible solutions to the testing problem

Relates to: https://gitlab.com/charts/helm.gitlab.io/issues/28

Plan:

Restore steps

The testing will basically be a testing script that gets called through the gitlab-ci in a separate job.

There are 2 main cases for the restore I would like the script to tackle.

1. A user with a fresh installed instance trying to restore from a backup that is uploaded to object storage.

2. A user with an instance that already has data and would like to clean up this data and replace it with a backup

For either cases we will do the following:

1. Upload the backup tar (contained in the repo) to minio.

2. Run the test cases

Test cases: run backup-utility --restore -t <backup-timestamp>

1. The restore completes successfully with no errors (exit 0)

2. Trying to hit <domain> works (returns 200)

3. Navigating into a restored repo works

4. Cloning the repo works

5. gitlab-runners are still registered (don't know how to do this for now, requires further investigation) which I know will fail because of https://gitlab.com/charts/gitlab/issues/466 upstream bug. I think we need to tackle this as soon as possible

6. Test plan needs to verify that previously uploaded issue attachments are loading

7. Previously ran CI job needs to show traces

8. Registry is serving previously uploaded images

We can then run another backup-utility --restore -t <another-timestamp> to use another backup to test being able to restore into a non-fresh install instance.

We can also run qa tests against this instance after a restore

I would like this script to be written in ruby may be using rspec or something that allows us to easily do assertions, expectations .. etc.

This scenario needs to run in reasonable amount of time, it doesn't make sense that the entire pipeline takes 10 mins while this takes more

Backup steps

All that will only cover the restore part.

To cover the backup part we will need to do a backup after restore and compare with the backup that we used to restore in the first place. Expectation should be that the only difference should be the backup-information.yml file.

We will also expect that the backup file is pushed correctly into object storage with the correct naming