Skip to content

Link to test screenshot does not work for tests with error instead of failure

Summary

I've been trying to use recently added (announced in gitlab 13.12) feature failed test screenshots in test report added in issue #216979 (closed)

I'm using PHP with library phpunit to generate junit test report. It works fine for tests that failed assertions, however it doesn't work for exceptions. The attachment is only extracted from junit for test cases with <failure> element, but not from those with <error> element.

Steps to reproduce

  1. In a CI job, generate junit.xml as artfacts.reports.junit that have image attached in a testcase with <error> element.
  2. Go to test_report of a pipeline, or a merge request.
  3. Click view details on test that have error instead of failure

Example Project

You can see here https://gitlab.com/trakos/screenshot-test/-/pipelines/308137499/test_report that attachment is not visible for App.Tests.EndToEndTest::testHomeException.

It works fine here https://gitlab.com/trakos/screenshot-test/-/pipelines/308141546/test_report - it's the same code, but I've added this lines go .gitlab-ci.yml:

    after_script:
        - sed -i 's/<error/<failure/g' junit.xml
        - sed -i 's/<\/error>/<\/failure>/g' junit.xml

It's an ugly workaround that might not be acceptable for bigger projects.

What is the current bug behavior?

For this junit.xml generated as artfacts.reports.junit:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="2" assertions="1" errors="1" warnings="0" failures="2" skipped="0" time="3.638328">
    <testsuite name="Project Test Suite" tests="2" assertions="1" errors="1" warnings="0" failures="1" skipped="0" time="3.638328">
      <testsuite name="App\Tests\EndToEndTest" file="/builds/trakos/screenshot-test/tests/EndToEndTest.php" tests="2" assertions="1" errors="1" warnings="0" failures="1" skipped="0" time="3.638328">
        <testcase name="testExample" class="App\Tests\EndToEndTest" classname="App.Tests.EndToEndTest" file="/builds/trakos/screenshot-test/tests/EndToEndTest.php" line="17" assertions="1" time="0.202766">
          <failure type="PHPUnit\Framework\ExpectationFailedException">App\Tests\EndToEndTest::testExample
Failed asserting that '' contains "example".

/builds/trakos/screenshot-test/vendor/symfony/panther/src/WebTestAssertionsTrait.php:96
/builds/trakos/screenshot-test/tests/EndToEndTest.php:22</failure>
          <system-out>[[ATTACHMENT|var/error-screenshots/2021-05-23_17-15-51_failure_App-Tests-EndToEndTest__testExample-0.png]]</system-out>
        </testcase>
        <testcase name="testHomeException" class="App\Tests\EndToEndTest" classname="App.Tests.EndToEndTest" file="/builds/trakos/screenshot-test/tests/EndToEndTest.php" line="25" assertions="0" time="0.369030">
          <error type="RuntimeException">App\Tests\EndToEndTest::testHomeException
RuntimeException: exception to force screenshot creation

/builds/trakos/screenshot-test/tests/EndToEndTest.php:30</error>
          <system-out>[[ATTACHMENT|var/error-screenshots/2021-05-23_17-15-52_error_App-Tests-EndToEndTest__testHomeException-0.png]]</system-out>
        </testcase>
      </testsuite>
    </testsuite>
  </testsuite>
</testsuites>

only the screenshot for the first <testcase> with <failure> element is linked to. The one with <error> is not linked to in test details.

What is the expected correct behavior?

I expected for both screenshots to be shown, both with <failure> element and with <error> element.

Relevant logs and/or screenshots

Before converting <error> to <failure>:

image

After converting <error> to <failure>:

image

Output of checks

Results of GitLab environment info

Results of GitLab application Check

This bug happens on GitLab.com

Possible fixes

The fix in code should be easy. It was added in this commit b8335525 . It looks like here: https://gitlab.com/gitlab-org/gitlab/-/blob/7a6e4f24d0d828500a7c7b6eb84973a1b4c35534/lib/gitlab/ci/parsers/test/junit.rb#L65 line attachment = attachment_path(data['system_out']) could be also added to elsif data.key?('error'). Based on MR !60528 (diffs) it's later displayed in the UI whenever it is returned in the endpoint, so that would be the only change required.

Alternatively, documentation should make clear that this limitation exist.

Edited by Piotr Stankowski