Improve test resources delete process
What does this MR do and why?
Describe in detail what your merge request does and why.
First half of QA team task https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1161 . The next part of the work is here gitlab-org/quality/pipeline-common!139 (merged)
- Add logic to write resources from failed suites to a separate file. I.e: append
failed
to file name. - Change test_resources_delete.rb to test_resources_handler.rb because it can now: delete resources, upload (failed) test resources to GCS bucket, and download and remove these files for deletion at a later time.
- Add new rake tasks and group them under a common namespace
test_resources
. - Using
google-cloud-storage
gem to connect to GCS since it is used in GCS documentation, should be easier for future maintenance.
My plan is once this is merged:
- Create another job under
clean up
stage on live environment pipelines calledupload-failed-test-resources
to upload files from failed suite run to GCS. - Create a dependent
delete-failed-test-resources
job that would be run in a scheduled pipeline (maybe on every other Sunday🤔 ) on these environments. This job will download the files from GCS first then run delete.
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
Success suite run will produce test-resources-<blah>.json
Failed suite run will produce failed-test-resources-<blah>.json
These failed files is uploaded to bucket under their environment accordingly
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.