Conan packages scoped to project-level
What does this MR do?
NOTE: This MR has over 1000 lines added. Don't panic! The reason is a 900+ line spec file was translated to a shared_examples file so it could be used in two different places. See the notes on that spec towards the bottom of this description.
🔍
A little background on what this is all about Instance-level packages
A package belongs to a project.
This MR implements project-level packages for the Conan package registry. Currently, GitLab has instance-level packages.
What this means is that right now, when you publish a package to GitLab, you use a remote
url that is common to the entire GitLab instance: conan remote add gitlab https://gitlab.com/api/v4/packages/conan
. When Conan sends the package to GitLab, we rely on the package name to tell us what project we should relate the package to. This means we have to force users to name their packages in a way that uniquely identifies the project.
The same goes for installing a package from the GitLab registry, GitLab only receives the name of the package, so it has to identify the project it's saved in in order to check permissions on the user making the request.
Project-level packages
If we add a set of routes that include the project ID, a user can set their remote with conan remote add gitlab https://gitlab.com/api/v4/projects/<project_id>/packages/conan
. This means when they publish or install a package, GitLab does not need to identify the project from the package name, but can find it directly from the id in the requests. When this is the case, we can allow users to name their packages anything they'd like!
⚙
Back to what this MR does This MR implements project-level packages by:
- Rename the current
conan_packages.rb
API file toconan_package_endpoints.rb
and turn it into anActiveSupport::Concern
module that can be included in other API files. - Create
conan_instance_packages.rb
andconan_project_packages.rb
and wrapConanPackageEndpoints
inside of each respective namespace. This is to allow us to use the same endpoints with different prefixes.
- Instance-level endpoints have the prefix
api/v4/packages/conan/v1/
- Project-level endpoints have the prefix
api/v4/projects/:id/packages/conan/v1/
- Rename
conan_packages_spec.rb
toconan_instance_packages_spec.rb
and createconan_project_packages_spec.rb
(a few notes on this decision below). - A few of the Conan API endpoints returns a set of URLs for the Conan client to use to upload or download the individual package files. We update the code in the
Conan::PackagePresenter
andConan::ApiHelpers
so it uses the correct Grape path helper to build the urls with the correct prefix (instance or project level). - Update the docs to describe how to use the project-level remote and update the naming restriction content.
Notes on the specs
Because we now have two sets of API endpoints that do essentially the same thing, but with different url prefixes, we can mostly use the existing instance-level tests for both. Here is the approach I took
- You cannot have dynamic data in the
describe
descriptions, so I decided to leave adescribe
for each endpoint in each spec file. - From there, I attempted to keep each endpoint as small as possible, using shared_examples so that both the project-level and instance-level run the same tests, but just feed them different URLs for the API calls.
- The only other major difference is that when testing for a "not found project", the project-level tests must base this on an invalid
project_id
param, and the instance-level tests must base this on an invalidpackage_username
param. - So although there is a lot of changes in the test code here, the
conan_packages_shared_examples.rb
andconan_packages_shared_context.rb
files don't really contain any new code, just code migrated from the previousconan_packages_spec.rb
file.
I'm happy to discuss if you have any questions here!
📷
Screenshots Uploading a package with a project level remote
$ conan remote add localproject http://localhost:3001/api/v4/projects/9/packages/conan
$ CONAN_LOGIN_USERNAME=root CONAN_PASSWORD=<redacted> conan upload Hello/0.1@testname/stable --all -r localproject
DEBUG :conan_api.py [163]: INIT: Using config '/Users/steveabrams/.conan/conan.conf' [2020-08-20 14:51:16,883]
DEBUG :tracer.py [155]: CONAN_API: upload(pattern=Hello/0.1@testname/stable,package=None,query=None,remote_name=localproject,all_packages=True,policy=None,confirm=False,retry=None,retry_wait=None,integrity_check=False) [2020-08-20 14:51:16,884]
Uploading to remote 'localproject':
Uploading Hello/0.1@testname/stable to remote 'localproject'
DEBUG :rest_client_common.py[160]: REST: ping: http://localhost:3001/api/v4/projects/9/packages/conan/v1/ping [2020-08-20 14:51:16,901]
DEBUG :rest_client.py [38]: REST: Cached capabilities for the remote: [] [2020-08-20 14:51:17,300]
DEBUG :rest_client_common.py[185]: REST: get: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/digest [2020-08-20 14:51:17,300]
DEBUG :rest_client_common.py[36]: REST ERROR: <class 'conans.errors.NotFoundException'> [2020-08-20 14:51:17,572]
DEBUG :rest_client_common.py[152]: REST: Check credentials: http://localhost:3001/api/v4/projects/9/packages/conan/v1/users/check_credentials [2020-08-20 14:51:17,572]
DEBUG :rest_client_common.py[185]: REST: get: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable [2020-08-20 14:51:17,887]
DEBUG :rest_client_common.py[179]: REST: post: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/upload_urls [2020-08-20 14:51:18,125]
Uploading conanfile.py: 100%|##########| 1.72k/1.72k [00:00<00:00, 872kB/s]
Uploading conanmanifest.txt: 100%|##########| 58.0/58.0 [00:00<00:00, 30.6kB/s]
DEBUG :rest_client_v1.py[161]: UPLOAD:
All uploaded! Total time: 2.153989315032959
[2020-08-20 14:51:20,528]
Uploaded conan recipe 'Hello/0.1@testname/stable' to 'localproject': http://localhost:3001/api/v4/projects/9/packages/conan
Uploading package 1/1: 103f6067a947f366ef91fc1b7da351c588d1827f to 'localproject'
DEBUG :uploader.py [353]: UPLOAD: Time remote_manager build_files_set : 0.001200 [2020-08-20 14:51:20,530]
DEBUG :rest_client_common.py[152]: REST: Check credentials: http://localhost:3001/api/v4/projects/9/packages/conan/v1/users/check_credentials [2020-08-20 14:51:20,532]
DEBUG :rest_client_common.py[185]: REST: get: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f [2020-08-20 14:51:20,942]
Requesting upload urls... DEBUG :rest_client_common.py[179]: REST: post: http://localhost:3001/api/v4/projects/9/packages/conan/v1/conans/Hello/0.1/testname/stable/packages/103f6067a947f366ef91fc1b7da351c588d1827f/upload_urls [2020-08-20 14:51:21,176]
Requesting upload urls...Done!
Uploading conan_package.tgz: 100%|##########| 2.08k/2.08k [00:00<00:00, 965kB/s]
Uploading conaninfo.txt: 100%|##########| 440/440 [00:00<00:00, 223kB/s]
Uploading conanmanifest.txt: 100%|##########| 158/158 [00:00<00:00, 76.9kB/s]
DEBUG :rest_client_v1.py[161]: UPLOAD:
All uploaded! Total time: 3.1885528564453125
[2020-08-20 14:51:24,618]
DEBUG :uploader.py [291]: UPLOAD: Time upload package: 4.089219 [2020-08-20 14:51:24,619]
DEBUG :uploader.py [300]: UPLOAD: Time uploader upload_package: 4.090056 [2020-08-20 14:51:24,619]
DEBUG :uploader.py [91]: UPLOAD: Time manager upload: 7.733696 [2020-08-20 14:51:24,620]
Screenshot of the package showing in the project
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides - [-] Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team
Related #11679 (closed)