Add config file class for Python Pip `requirements.txt`
What does this MR do and why?
Background
The Repository X-Ray is a tool that searches for dependency manager configuration or lock files and then extracts a list of libraries from the content.
In #474306 (comment 2025085630), we decided to migrate the Repository X-Ray functionality into the GitLab Rails monolith. This gives us two main benefits: (i) it will eventually allow us to run the service outside of the CI pipeline, and (ii) we can maintain the parsing logic centrally so that other domains can utilize it.
Base MR
In !162313 (merged), we introduced the ConfigFiles::Base
class where the intention is for each dependency manager config file type to be represented by a child class. The child class contains the parsing logic to extract a list of libraries and their versions from the file content.
This MR
In this MR, we introduce the PythonPip
config file class that parses requirements.txt
for the language Python
. Please note that it does not currently supported nested requirements files; this will be addressed in #491800 (closed).
For references to the existing parsing logic in the Repository X-Ray repo, see:
- https://gitlab.com/gitlab-org/code-creation/repository-x-ray/-/blob/6634564818dffdeac938014a190b0c604b1da6e0/internal/scanner/scanner.go#L232
- https://gitlab.com/gitlab-org/code-creation/repository-x-ray/-/blob/6634564818dffdeac938014a190b0c604b1da6e0/internal/deps/deps.go#L112
Resolves part of #476177 (closed).
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
-
Enable the feature flag:
Feature.enable(:ai_enable_internal_repository_xray_service)
-
Create a new blank project and add the following file to the repo:
subdir/requirements.txt
:
requests>=2.0,<3.0 # Version range
numpy==1.26.4 # Exact version match
python_dateutil>=2.5.3
fastapi-health!=0.3.0
# New supported formats
pytest >= 2.6.4 ; python_version < '3.8'
openpyxl == 3.1.2
urllib3 @ https://github.com/path/main.zip
requests [security] >= 2.8.1, == 2.8.*
# Nested requirement files currently not supported
-r other_requirements.txt
# Other options
-i https://pypi.org/simple
--python-version 3
--no-clean
-e .
- Observe that the X-ray report for the config file is recorded:
project = Project.last # Should be the project you created earlier
project.reload.xray_reports
Related to #476177 (closed)