Add option to retry failed helm deployment

What does this MR do and why?

Adds configurable retry option for failed deployment.

Part of: gitlab-org/quality/quality-engineering/team-tasks#3067
Closes: gitlab-org/quality/quality-engineering/team-tasks#3068 (closed)

Note

Retry is not set in the pipeline configuration because we still want to run experiment of running this in merge requests and want to capture as many failures as possible: Run experiment with running 'test-on-cng' in me... (gitlab-org/quality/quality-engineering/team-tasks#3049 - closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

[✖] running helm deployment ... failed
Upgrading helm release 'gitlab' in namespace 'gitlab'
Deployment failed, retrying...
Error: Command 'helm upgrade --install gitlab gitlab/gitlab --namespace gitlab --timeout 1m --values - --wait --set gitlab.webservice.workhorse.tag=39e91ce799348671b1385fe051c78404c85cea81' failed!
Error: UPGRADE FAILED: context deadline exceeded

Upgrading helm release 'gitlab' in namespace 'gitlab'
Helm upgrade failed!
For more information on troubleshooting failures, see: 'https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/gems/gitlab-cng?ref_type=heads#troubleshooting'
Following events of Warning type present in cluster:
...