E2E cleanup: fix deletion errors and fail at the end
What does this MR do and why?
Retry on 502 response errors to prevent errors in parse_body
in wait_for_resource_deletion
so it doesn't stop the cleanup script in the middle of a run.
Don't log failure if resource returns 404 when trying to delete as a 404 is the end goal of deletion.
We then need to fail the job after the script is done if there were any errors. This allows for automatic retry as well.
Example error of script stopping in the middle: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/13826859
Example error of 404 even though we want the resource to be deleted: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/13860896
Example error of job succeeding even though there were errors: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/13859985
Issue: gitlab-org/quality/quality-engineering/team-tasks#2686 (closed)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.