Skip to content

Test History MVC - Full Unit Test Report

Release notes

When you are reviewing the failed tests in a pipeline it can be hard to tell if a failing test is a legitimate failure that needs investigation or a flaky test that will pass on the next execution.

The next iteration of the repeat failed test counter is adding a repeat failed test counter to the unit test report. Now you can see how often a test has failed on the default branch without needing to have a Merge Request associated with the test execution.

https://docs.gitlab.com/ee/ci/unit_test_reports.html#number-of-recent-failures

Problem to solve

Developers do not have an easy way to get the history of if a test has recently been passing, failing or skipped as they research a test failure in the context of a test run in a Pipeline.

Without this context it is hard to point at why the build is feeling/getting slower and what test cases may to be blame for that. Without that context it's not hard to imagine Delaney and Sasha starting to skip tests and not write tests for new code.

Intended users

User experience goal

The user should be able to use the UI with GitLab to see if a test has passed or failed more often over the last N executions in the JUnit report.

Proposal

This is an MVC proposal to measure if there is interest / value in this feature. If there is we will likely re-architect this to be more scalable and functional going forward.

  • Display a tooltip / column on the Unit Test report for the data (see mock below but designs should be done by @jj-ramirez ).
  • Track page views of the Unit Suite page as a snowplow event we can display in the group dashboard.

Backend

The recent_failures relies on pre-loaded data which was only done on the MR widget's test report endpoint. So we need to do something similar for the endpoint used on the pipelines test report page.

Design - See below.

Further details

Before investing in time to architect a real solution for this to start on the epic for Testing History for MRs and Pipelines or even the MVC for Test History for Projects we will validate that there is value and move the feature forward with this MVC. We fully expect not to re-use code and maybe interface built as part of this issue as we learn what is valuable.

Permissions and Security

Documentation

We should update the documentation to reflect that this message can appear on both the MR widget and the test report summary pages.

Availability & Testing

@zeffmorgan @rickywiens @samdbeckham - Can you ponder on some performance considerations for this section and edit accordingly please?

What does success look like, and how can we measure that?

Acceptance Criteria

  • Test execution data is tracked for all failed tests and displayed on the Pipeline -> Unit Test Report
  • Pipeline -> Unit Test report can still load as quickly as it does today
  • Page Views for Unit Test report are tracked - A separate issue has been created for this.
GIVEN a junit artifact is uploaded during an MR pipeline run
GIVEN there is at least one failed test in the junit report artifact
WHEN you visit the pipeline page
WHEN you click on the unit test report tab
THEN there exists a tooltip beside the failing test indicating how many times its' failed recently

Measures of Success

  • This feature will be successful if we collect any feedback from internal and external users within 45 days of launch to inform the next iteration.

What is the type of buyer?

The buyer for this is the developer who wants to see how a test is behaving previously to inform their current MR.

Is this a cross-stage feature?

No.

Links / references

Feedback for this MVC will be collected in the feedback issue.

Edited by Ricky Wiens