Skip to content

Implement Repository X-Ray CI pipelines that are triggered automatically

Summary

Currently, to run the Repository X-Ray in a project, a user must copy the following config and paste it into their Pipeline config Yaml file.

xray:
  stage: build
  image: registry.gitlab.com/gitlab-org/code-creation/repository-x-ray:latest
  allow_failure: true
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  variables:
    OUTPUT_DIR: reports
  script:
    - x-ray-scan -p "$CI_PROJECT_DIR" -o "$OUTPUT_DIR"
  artifacts:
    reports:
      repository_xray: "$OUTPUT_DIR/*/*.json"

The above approach has two main customer concerns:

  1. It consumes CI minutes.
  2. It shouldn't be anything that users have to configure and maintain.

In Spike: Investigate possibility to run X-Ray bin... (#467357 - closed), we investigated alternative approaches to address the above concerns. We decided on one that involves running the X-Ray job in a standalone pipeline that is exempt from CI minutes (#467357 (comment 1997678846)).

A POC MR (!159818 (closed)) was created to verify the approach. And we decided to proceed. This issue is to keep track of the effort to implement this new approach.

Proposal

Implement the POC MR: Draft: POC - Run X-Ray automatically in standal... (!159818 - closed)

The approach is as follows:

  • The X-Ray job runs in a special CI pipeline that is triggered when a new commit is pushed to the project's default branch.
  • This feature is enabled via a project setting.
  • The number of X-Ray pipelines are minimized with a self-limiting behaviour as follows:
  #   - We use an exclusive lease guard to avoid creating duplicate RX pipelines simultaneously.
  #   - We check the status of the last RX pipeline to ensure we only have one RX pipeline alive at a time.
  #   - If the last RX pipeline was already created for the latest commit, a new one is not created.
  #   - The worker is rescheduled if the lease is taken or if the last RX pipeline is still alive.

MR Implementation Breakdown

Description MR / Link
Add a new CI pipeline source :repository_xray; exempt them from the pipeline rate limit and CI minutes !159986 (closed)
Add a new service that creates the Repository X-Ray pipeline, with self-limiting behaviour
Add a new worker to execute the RX pipeline service, and set it to be enqueued after a new commit is pushed to the project's default branch. Feature flag: repository_xray_pipelines
Update the frontend to render a repository x-ray badge in the Pipelines view

Update [2024-07-24]

Unfortunately due to complications with CI pipelines explained in !159818 (comment 2014396807), it appears that the automated X-Ray pipeline POC is not a viable interim approach.

We should further investigate integrating with groupcomposition analysis dependency scanning per #467357 (comment 2004198368). I have opened a separate issue for this: Spike: Investigate replacing or integrating Rep... (#474306 - closed).

Edited by Leaminn Ma