Add logic to retry if pipeline doesnt finish

What does this MR do and why?

Describe in detail what your merge request does and why.

I'm not sure why sometimes pipeline stuck in pending state for more than 1 minute that causes run_pipeline_with_manual_jobs_spec.rb to fail - failure issue #359209 (closed). Since I can't reproduce locally, I'm adding a retry logic for when that happens as well as try to grab some job that I think could be that problems (they come with failure_reason attribute) to see if that would help with debugging.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #359209 (closed)