Add PyPI package requests forward
🎟 Context
The npm package registry has a feature called the package request forward. It can be summarized as:
When a package is requested to the GitLab package registry and the package doesn't exist in the accessed resource, forward that request to the official registry.
This behavior is controlled with an application setting which is enabled by default on gitlab.com.
PyPI users voiced their interest on this feature. That's issue #233413 (closed).
🔍 What does this MR do?
This MR is an exact copy of the npm package request forward for pypi packages.
- Update the pypi finder so that it can't raise an
ActiveRecord::NotFound
error. - Update the metadata API endpoints so that if the given package name is not found, the request gets redirected.
- Update the dependency proxy helpers to support different package types.
- Add an application setting that gates this behavior.
- Update/add all related specs.
A small word on the EE side. For some reason, this application setting is marked as premium
. See https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#npm-forwarding. The issue is that package features have been moved to Core. As such, the API endpoints when using the package forward function, they don't check the license features.
In other words, package request forward is in the EE side but it should be moved to Core. This is way outside of this MR and we opened #337862 for that.
For consistency (with the existing similar npm feature), the changes here are in EE but keep in mind that we're going to remove those very soon (currently scheduled for %14.4)
📸 Screenshots or Screencasts (strongly suggested)
Assume that we have a project with a single pypi package:
Project.find(38).packages.pypi.map(&:name)
=> ["pypibananas"]
Project level
Let's try to pull it using the project level instance:
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps pypibananas
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
Collecting pypibananas
Downloading http://gdk.test:8000/api/v4/projects/38/packages/pypi/files/26daf30525a79d6c75dd93e16f469acc050ed315d90434b09e218345ba671b3c/pypibananas-1.3.7-py3-none-any.whl (1.4 kB)
Installing collected packages: pypibananas
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pypibananas-1.3.7
It works
Now with the application setting enabled, let's try to pull a package that is not in the project:
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
Collecting pytmi
Using cached pytmi-0.2.0-py3-none-any.whl (6.8 kB)
Installing collected packages: pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pytmi-0.2.0
It works
Let's now disable the application setting and try again:
$ pip uninstall pytmi
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement pytmi (from versions: none)
ERROR: No matching distribution found for pytmi
It works
Lastly, with the application enabled, let's try to pull a package that doesn't exist anywhere (project or the official registry):
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/projects/38/packages/pypi/simple --no-deps idonotexist
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/projects/38/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement idonotexist (from versions: none)
ERROR: No matching distribution found for idonotexist
Package not found
Group level
Let's run the same cases but with the group level endpoint.
Pulling the existing package:
$ pip uninstall pypibananas
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps pypibananas
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
Collecting pypibananas
Downloading http://gdk.test:8000/api/v4/groups/113/-/packages/pypi/files/b6c1510e1289a906bfaa5b551ea95ae6570f87bc0b18ab753de6bf38b9cbc1c7/pypibananas-1.3.7-py3-none-any.whl (1.4 kB)
Installing collected packages: pypibananas
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pypibananas-1.3.7
It works
Now with the application setting enabled, let's try to pull a package that is not in the group:
$ pip uninstall pytmi
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
Collecting pytmi
Using cached pytmi-0.2.0-py3-none-any.whl (6.8 kB)
Installing collected packages: pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pytmi-0.2.0
Works
Let's now disable the application setting and try again:
$ pip uninstall pytmi
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps pytmi
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement pytmi (from versions: none)
ERROR: No matching distribution found for pytmi
Package not found
Lastly, with the application enabled, let's try to pull a package that doesn't exist anywhere (project or the official registry):
$ pip install --trusted-host gdk.test --index-url http://root:XXXX@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple --no-deps idonotexist
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Looking in indexes: http://root:****@gdk.test:8000/api/v4/groups/113/-/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement idonotexist (from versions: none)
ERROR: No matching distribution found for idonotexist
Package not found
⚗ How to setup and validate locally (strongly suggested)
- Have a personal access token with the api scope ready.
- Enable or disable the application setting in:
/admin/application_settings/ci_cd#js-package-settings
. -
Install
pip
and try the commands above.
📐 Does this MR meet the acceptance criteria?
Conformity
-
I have included changelog trailers, or none are needed. (Does this MR need a changelog?) -
I have added/updated documentation, or it's not needed. (Is documentation required?) -
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.
Availability and Testing
-
I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.) - [-] I have tested this MR in all supported browsers, or it's not needed.
- [-] I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.
Security
Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team
💽 Database review
Up
$ rails db:migrate
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: migrating
-- add_column(:application_settings, :pypi_package_requests_forwarding, :boolean, {:default=>true, :null=>false})
-> 0.0041s
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: migrated (0.0120s)
Down
$ rails db:rollback
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: reverting
-- remove_column(:application_settings, :pypi_package_requests_forwarding)
-> 0.0059s
== 20210806152104 AddPypiPackageRequestsForwardingToApplicationSettings: reverted (0.0182s)