Simplify test and add wait and retry logics

What does this MR do?

To address failures:

• #300914 (closed)
• #301110 (closed)
• #320849 (closed)
• #321585 (closed)

Also to simplify/optimize this test. One of the thing to notice is I noticed a new behavior (merge train could failed if starts right after MR pipeline just finishes, reported here #321001 (closed)) so I add a work around to help test pass until the issue is sorted out , hopefully during re-architect #300042 (closed).

Does this MR meet the acceptance criteria?

Conformity

• [-] Changelog entry
• [-] Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• [-] Database guides
• Separation of EE specific content