Deprecate X-Ray CI job and execute X-Ray automatically in a worker

What does this MR do and why?

Background

The Repository X-Ray is a tool that searches for dependency manager configuration/lock files and then extracts a list of libraries from the content.

Currently, the X-Ray scan is run in a CI job that must be manually configured in the CI pipeline. In #474306 (comment 2025085630), we decided to migrate the Repository X-Ray functionality into the GitLab Rails monolith. This gives us two main benefits: (i) it will eventually allow us to run the service outside of the CI pipeline, and (ii) we can maintain the parsing logic centrally so that other domains can utilize it.

Previous MR

In !164404 (merged), we merged a new service Ai::RepositoryXray::ScanDependenciesService, which utilizes Ai::Context::Dependencies::ConfigFileParser to extract the config files from a repository. Then it saves the parsed data to the xray_reports table.

This MR

In this MR, we introduce the worker Ai::RepositoryXray::ScanDependenciesWorker which runs the Xray service and is triggered upon a new commit pushed to the project's default branch. These changes are made behind a feature flag which does two things:

1. Enables the new worker.
2. Disables consuming the artifacts (Xray reports) from the X-ray CI job.

Feature flag: ai_enable_internal_repository_xray_service, Roll-out issue: #483928 (closed)

Additionally, some minor refactors are included in this MR:

• Changed Ai::RepositoryXray::ScanDependenciesService to reschedule the worker and return an error response if it cannot obtain a lease.
• Updated Ai::RepositoryXray::ScanDependenciesService to support the case when there are no valid config files extracted.

Resolves #476180 (closed).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

The X-ray reports remain largely the same before and after enabling this feature. The minor differences, such as scanner version, brackets, and lack of description value, are inconsequential. The "after" format is simply a cleaner version.

Before: X-ray report content saved from the CI job	After: X-ray report content saved from the new in-monolith service

How to set up and validate locally

1. Ensure you have GitLab Duo Code Suggestions enabled on your gdk.
2. Start your gdk and enable the feature flag: Feature.enable(:ai_enable_internal_repository_xray_service).
3. Create a new project. In the project's root, add the following file.

go.mod

require (
  golang.org/x/mod v0.15.0
  golang.org/pmezard/go-difflib v1.0.0
)

4. In the rails console, observe that the expected X-ray report has been saved.

project = Project.last # This should be the new project you created
project.xray_reports

5. Modify go.mod with the following and commit the change.

require (
  golang.org/mylib1 v0.15.0
  github.com/mylib2 v1.0.0
  github.com/mylib3 v1.0.1
)

6. Observe that the expected X-ray report has been updated.

project.reload.xray_reports

Related to #476180 (closed)