Investigate "Known Issues" feature of SQUAD (mark tests as "allowed to fail")
Issue Background
SQUAD offers the possibility to mark tests as has_known_issues
. A test also features the field issues
.
We would like to know if this can be used to marks tests, e.g. to be able to identify them as unimportant or known to fail.
Issue description
Find out how to add a known issues and what it actually means for the SQUAD reporting. This should be documented somewhere, possibly together with a script or tool, that shows how to mark known issues.
Acceptance criteria
-
It is documented how to create a known issue -
A small script is provided, that shows how to create a new known issue -
The effect on the test reports of known issues has been clarified and documented -
It is clarified if the known issues feature in SQUAD can be used to mark tests as allowed to fail
Outcome
- Known issues can be created using the administration panel:
- Required fields:
- test suite and test name:
<suite-name>/<test-name>
, e.g.spectre-meltdown-checker-test/CVE-2018-3640
) - One or more group/project/environment combinations:
<group-name>/<project-name> - <environment-name>
, e.g.linux-cip/linux-6.1.y-cip - arm_renesas_shmobile_defconfig_r8a7743-iwg20d-q7
- test suite and test name:
- The known issues tab gives an overview of tests with known issues (see e.g. here)
- Known issues are not registered at previously reported tests. E.g. this test has known issues, but is not registered.
- Test result reporting, after a known issue has been created, are marked as
xfail
. See for example this result
- Required fields:
Edited by Sietze van Buuren