JUnit integration reports skipped tests as passed
Summary
JUnit integration reports skipped tests as passed
Problem to solve
Sasha is writing unit tests along with code with results in a format GitLab can consume and report on about what passed and failed. This is great if all tests are run but Sasha is starting to write more complex tests that are sometimes skipped based on previous test results or other external factors to the test suite. Today those skipped tests are showing up as successful test executions in the GitLab reports, an incorrect signal to Sasha and the team about the reliability of their code.
Intended users
- Sasha (Software Developer) - who is seeing test results in pipeline results and merge requests.
- Simone (Software Engineer in Test) - Who is validating test coverage and effectiveness of those tests.
Further details
This enhances the JUnit reports which already has a column for skipped but is not being populated today.
A future enhancement could be to update the Test Summary Widget to include additional information about the tests skipped on the head branch compared to results on the base branch.
Steps to reproduce
-
Go to the following repository: https://gitlab.com/marknelson/junit-issue-reproducer
-
Run the CI/CD pipeline in that repository, or just look at this run: https://gitlab.com/marknelson/junit-issue-reproducer/pipelines/123638823
-
Look at the job output: https://gitlab.com/marknelson/junit-issue-reproducer/-/jobs/460993965
-
Notice, at the end it reports the test results:
Ran 1 of 2 Specs in 0.003 seconds SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 1 Skipped PASS
-
The JUnit file produced has the same info, for reference:
<?xml version="1.0" encoding="UTF-8"?> <testsuite name="My Test Suite" tests="1" failures="0" errors="0" time="0"> <testcase name="Tests can pass" classname="My Test Suite" time="5.28e-06"></testcase> <testcase name="Tests can be skipped" classname="My Test Suite" time="0.00021871"> <skipped></skipped> </testcase> </testsuite>
-
Now look at the tests tab in the pipeline: https://gitlab.com/marknelson/junit-issue-reproducer/pipelines/123638823/test_report
-
Notice it reports 2 passed, 0 skipped. It should report 1 passed, 1 skipped.
Example Project
see above
What is the current behavior?
GitLab CI pipeline test report counts skipped tests as "passed".
What is the expected new behavior?
GitLab CI pipeline test report should count skipped tests as "skipped".
Relevant logs and/or screenshots
see above
Output of checks
This bug happens on GitLab.com (and on our private installation too)
Possible fixes
This seems to be a spec/test for this feature, it does not seem to inlcude a test for skipped tests: https://gitlab.com/gitlab-org/gitlab/-/blob/acc05479b1723f8804579548d8d3bde2478c18fd/spec/lib/gitlab/ci/parsers/test/junit_spec.rb
This code seems to be where it creates the report - it seems to only handle pass, fail and error, not skipped. https://gitlab.com/gitlab-org/gitlab/-/blob/acc05479b1723f8804579548d8d3bde2478c18fd/lib/gitlab/ci/parsers/test/junit.rb#L48
All of the serializers and Test(Reports|Suites|Case)
classes seems to handle skipped
already, and we're just not correctly parsing out the skipped
status in Ci::Parsers::Test:Junit#create_test_case
. Adding some failure-affirming specs there and then adding handling for skipped cases should do the trick.
Further details
This enhances the JUnit reports which already has a column for skipped but is not being populated today.
A future enhancement will be to update the Test Summary Widget to include additional information about the tests skipped on the head branch compared to results on the base branch.