Skip to content

Include test results within Release Evidence

Problem to solve

Allow users to create links between issues and CI Tests.

Assuming that an issue is an all-inclusive resolution of a single problem, this would enable a test to be added to the CI pipeline to ensure the problem was resolved prior to closing the issue.

Intended users

Further details

The ability for a user to link an issue with a CI Test would enable test tracing at a very basic level. There is some thought that this could be an MVC for Requirements Management.

  • Consider the artifacts generated out of the test stage
  • Reports from testing results
    • Consumable report/record about the code and baseline from where you started and stopped
    • What does that evidence need to look like?
  • Persisting testing statuses per linked MR in release evidence JSON file
    • Latest job artifact persists - might make this easier
  • Code climate spec and JUnit Format - investigate if we can persist this information?
  • Sonar cube jobs - investigate the artifact created and then uploaded to the release as evidence

Proposal

  • Add testing results to MR IDs in release evidence JSON

MVC for this feature is:

  • feature is triggered by Evidence collection
  • prevent expiration of artifacts associated with pipeline which creates the Evidence are marked as non-expiration. This could be done via the Keep Artifacts endpoint POST /projects/:id/jobs/:job_id/artifacts/keep. This would capture all artifacts associated with the pipeline.
  • identify all the jobs in the pipeline using the List pipeline jobs GET /projects/:id/pipelines/:pipeline_id/jobs
  • persist the list of artifacts for those jobs in the Evidence hash, using the Get Job Artifacts endpoint GET /projects/:id/pipelines/:pipeline_id/jobs

This would, in this iteration, prevent any issue with access control, as links-only are persisted into the Evidence hash.

Permissions and Security

  • Testing data within release evidence should be maintainer & auditor role only
  • Testing data is based on job results, consideration on access and investigation into confidential issues and related MRs

Documentation

Documentation would need to be updated to describe the process of creating a link between an issue and a CI test. It would also be beneficial to provide users a use case for how this could benefit their testing coverage.

Testing

Testing needs will need to be determined once discovery is complete and a better understanding of how this capability is implemented.

What does success look like, and how can we measure that?

  • Users would be adding test results to the Release Evidence - we need to track the additions of artifacts to release evidence

What is the type of buyer?

The purpose of linking issues and CI testing would be to provide compliance data. Therefore, this feature would most appeal to Executive level buyers.

  • Ultimate

Future considerations

  • Code quality as a gate in both auto deployment and runbooks
  • Web performance
  • Integration
  • Emulate Security practices
  • Test case management - how do we consider those against code coverage and velocity

Links / references

Recording to discussion

Anchor to Documentation: https://docs.gitlab.com/ee/user/project/releases/#release-evidence-test-results

Edited by Jackie Porter