Approach to E2E tests for the Secure stage functionality of GitLab

Currently, we don't have any system or E2E tests for the functionality of the ~"devops:secure" stage pages (Group Security Dashboard, Project Security Dashboard, Merge Request Security Reports, Pipeline Security Reports).

This issue is to discuss possible approaches from the perspective of the velocity/quality balance and to agree on the right approach.

Secure test plan

Introduction

This is a test plan to backfill basic e2e tests for the current secure features:

1. Merge request security reports
2. Group security dashboard
3. Auto-remediation process
4. Project security dashboard
5. Pipeline security reports

The goal is to check that the reports are shown for each configured feature after triggering a pipeline that has security products feature enabled. Additionally, we should verify that auto-remediation works to remove vulnerabilities from the merge request.

Scope

• This plan covers ensuring that the various reports and dashboards above are accessible

ACC Matrix

	Secure	Responsive	Intuitive	Reliable
Groups				2
Project				2
MRs		1	1	3
CI/CD				2

Capabilities

• Merge request is
  • Intuitive
    • When dependency scanning reveals vulnerabilities we can auto-remediate, it should be easy to trigger the auto-remediation process
  • Responsive
    • When the auto-remediation process is triggered, it should create the patch file immediately
  • Reliable
    • Once the pipeline for a merge request commit finishes, the merge request security report should be available
    • The merge request security report should be accurate
    • When the auto-remediation process is triggered and the pipeline finishes, the vulnerabilities should be gone
• Group is
  • Reliable
    • Once the pipeline for a merge request commit finishes, the group security dashboard should be available
    • The group security dashboard should be accurate
• Project is
  • Reliable
    • Once the pipeline for a merge request commit finishes, the project security dashboard should be available
    • The project security dashboard should be accurate
• CI/CD is
  • Reliable
    • Once the pipeline for a merge request commit finishes, the pipeline security report should be available
    • The pipeline security report should be accurate

Test Plan

Automated end-to-end tests

Scenario 1 Viewing a merge request security report

• Go to a project
• Trigger a pipeline that has the security products feature enabled
• Once the pipeline finishes, check the merge request security report is present and correct

Scenario 2 Viewing the projgroupect security dashboard

• Go to a project
• Trigger a pipeline that has the security products feature enabled
• Once the pipeline finishes, check the group security dashboard is updated and correct

Scenario 3 Viewing a merge request security report

• Go to a project
• Trigger a pipeline that has the security products feature enabled
• Once the pipeline finishes, trigger auto-remediation for the merge request
• Once the pipeline finishes, check the merge request security report is present and correct

Scenario 4 Viewing the project security dashboard

• Go to a project
• Trigger a pipeline that has the security products feature enabled
• Once the pipeline finishes, check the project security dashboard is updated and correct

Scenario 5 Viewing a pipeline security report

• Go to a project
• Trigger a pipeline that has the security products feature enabled
• Once the pipeline finishes, check the pipeline security report is present and correct

/cc @gl-quality This is my first test plan using the ACC format, so whatever feedback you have would be welcome!