Windows tests - ensure that nothing new starts to fail
We're executing our CI tests on two Windows machines (1803 and 1809). But currently all windows tests are marked as allow_failure: true
. When we've configured machines and tests we haven't time to go through all of the failures and decide which ones are caused by tests problems (most of them was created with *nix in mind) and which are real software failures.
Since we've recently added docker-windows
executor, which is our next big step into Windows OS ecosystem, we should start ensuring that we have no failures on Windows.
As a first iteration the idea is to:
- collect all failures that are present currently in tests executed on Windows and commit the list into the repository,
- leave the windows tests jobs as allowed to fail,
- in the coverage stage add a job that will:
- get all failures form previous tests executed on Windows,
- check the list against the list committed list,
- fail the pipeline if there will be any new failing test.
Next we can start going through the list of the failing tests and fixing them 1-by-1, each time removing the fixed test from the list. After removing the last one we will remove this special job and set tests executed on Windows to fail the pipeline in case of a failure.