Draft: POC - Run X-Ray automatically in standalone pipeline, exempt from CI minutes

What does this MR do and why?

This is the POC for #467357 (closed).

Background

Currently, to run the Repository X-Ray in a project, a user must copy the following config and paste it into their Pipeline config Yaml file.

xray:
  stage: build
  image: registry.gitlab.com/gitlab-org/code-creation/repository-x-ray:latest
  allow_failure: true
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  variables:
    OUTPUT_DIR: reports
  script:
    - x-ray-scan -p "$CI_PROJECT_DIR" -o "$OUTPUT_DIR"
  artifacts:
    reports:
      repository_xray: "$OUTPUT_DIR/*/*.json"

The above approach has two main customer concerns:

1. It consumes CI minutes.
2. It shouldn't be anything that users have to configure and maintain.

This MR

In this MR, we aim to solve both concerns by running the X-Ray scan in a separate pipeline that is triggered upon a new pushed commit to the project's default branch. The customer would simply need to enable Duo features on their project.

Note: RX = Repository X-Ray

Technical summary

• To avoid overloading the project with too many "extra" pipelines for the X-Ray scan, a self-limiting behaviour was implemented with a new worker and service: CreateRepositoryXrayPipelineWorker and CreateRepositoryXrayPipelineService
  • It is configured so that only one Repository X-Ray pipeline can run at a time per project.
  • Redundant pipelines are avoided by checking if the last RX pipeline already ran for the latest commit.
  • By employing a worker, we are able to utilize Sidekiq's queueing behaviour. Furthermore, parallel processing is prevented by using an exclusive lease guard in the service. Both of these tools help us minimize the number of times we have to create a new pipeline for the X-Ray scan.
• Since the RX pipeline is self-limiting, it is exempt from the pipeline rate throttling. And, per our objectives, the RX job is exempt from consuming CI minutes.
  • By running the job alone in a separate pipeline, we're able to identify the pipeline source :repository_xray and use it as an indicator for this exemption.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Even if multiple commits are pushed to the default branch within a small timeframe, a new Repository X-Ray pipeline is not created for each push nor is the existing one canceled.

In the case below, I pushed 5 consecutive commits to the main branch. Observe that only two Repository X-Ray pipelines were triggered. The first was triggered for the first commit. After it completed, one more Repository X-Ray pipeline was triggered to ensure the X-Ray report is updated with the latest commit.

Pipelines view for main branch

How to set up and validate locally

1. Ensure you have GitLab Duo Code Suggestions enabled on your gdk.
2. Ensure you have Gitlab Runner set up with your local gdk.
3. Start your gdk and create a new blank project.
4. In the project's root, add the following file. (This of course isn't a proper Gemfile.lock; it just has what we need for the Repository Xray to scan.)

Gemfile.lock

DEPENDENCIES
  acme-client (~> 2.0)
  bcrypt (~> 3.1, >= 3.1.14)
  capybara (~> 3.40)

5. Now let's commit a simple change to the project's main branch. In the project, go to Build > Pipeline editor, click "Configure pipeline", and replace the file contents with the following.

test-job:
  script: echo

Commit the change above. A new pipeline should execute automatically and run the test job.

6. Observe that a second pipeline is triggered with the badge repository x-ray.

7. After the repository x-ray pipeline is complete, you can check that the scan performed as expected by running this in your Rails console:

project = Project.last # This should be the newest project you just created
project.xray_reports

8. You can test that multiple (>2) consecutive commits to the main branch do not cause just as many repository x-ray pipelines to get triggered. Only one should be running at a time.

NOTE: It may take up to 1 minute for the next Repository X-Ray pipeline to trigger since there is an intentional rescheduling delay.

9. To test the CI minutes consumed, edit ee/app/services/ai/create_repository_xray_pipeline_service.rb and comment out the redundancy check (lines 39-41). Restart your gdk.

10. Then get the current CI minutes consumed from the Rails console as below. Take note of the number.

project = Project.last
Ci::Minutes::NamespaceMonthlyUsage.for_namespace(project.namespace)

11. Trigger the Repository X-Ray pipeline manually by running the following:

Ai::CreateRepositoryXrayPipelineService.new(project).execute

11. After the new RX pipeline completes, check the CI minutes again and observe that it has not increased.

Related to #467357 (closed)