Allow review-qa-all to fail when ~"pipeline:run-review-app" is presented
This is implemented in !88954 (merged)
Original title: Set expectations for pipeline:run-review-app label and document it
Background
While reviewing !87221 (merged) where pipeline:run-review-app
label was applied, I noticed that downstream review-qa-all
job was blocking - it looks like this is intentional but I couldn't find an explanation why. As opposed to review-qa-smoke
and review-qa-reliable
that we aim at keeping stable at all times, review-qa-all
job runs the full suite of E2Es and occasionally has unrelated flaky failures or known failures from master under investigation. When Review App is deployed in MRs automatically, that job is allowed to fail.
In the above mentioned MR, there were several unrelated failures in the jobs and we had to remove the label in order to get the MR merged. There is not much documentation regarding the label in the docs and it wasn't immediately clear if it is the right thing to do.
Proposal
-
We make
review-qa-all
allowed to fail. Pros: it doesn't block the MR from merging. Cons: any failures will be less visible and can be missed in case of MWPS. -
We keep
review-qa-all
blocking. Please note that there are certain tests that do not run well against Review Apps that might need to be modified/excluded from the job. Even after that there might still be occasional flaky failures/known failures from master present, so there will be situations where the author will have to remove the label to merge the MR.