Skip to content

Group level PyPi Index for Python modules published across multiple projects

Problem to solve

GitLab 12.10 introduced the GitLab PyPi Repository for storing Python modules in the GitLab Package Registry. I've been able to try this out successfully. But, I find the workflow really hard to manage when there is more than one project.

For example, I have two projects that have each published Python modules to their respective registries. Now, I want to use these published modules in a third project. I have to configure a separate repository or specify a separate index-url for each source project, because the repository endpoints are project specific.

Here's the Registry Setup verbiage provided in one of the projects. Note that the repository is specific to project ID 898.

If you haven't already done so, you will need to add the below to your .pypirc file.

[gitlab]
repository = https://gitlab.transzap.com/api/v4/projects/898/packages/pypi
username = __token__
password = <your personal access token>

For more information on the PyPi registry, see the documentation.

Unlike the Docker container registry where users log in to a single point (registry.gitlab.com or equivalent) and then have access to images across multiple GitLab projects subject to role permissions, the Python PyPi repositories have individual access points per GitLab project.

The per-project repository endpoint is really burdensome to manage when I want to use modules in more than one GitLab project.

Intended users

User experience goal

It should be possible to pull Python modules from multiple projects without specifying an index per project. Otherwise, module users have to know which project each module comes from by project ID, and manually manage a potentially long list of index URLs.

Proposal

I would like a common repository endpoint that allows me to pull Python modules from multiple projects. This could be instance wide like the Docker registry, or at the group level.

Further details

An install of three modules from three projects in the same group looks like this right now.

pip install \
    --extra-index-url https://$GITLAB_PYPI_INDEX_USERNAME:$GITLAB_PYPI_INDEX_PASSWORD@gitlab.transzap.com/api/v4/projects/897/packages/pypi/simple \
    --extra-index-url https://$GITLAB_PYPI_INDEX_USERNAME:$GITLAB_PYPI_INDEX_PASSWORD@gitlab.transzap.com/api/v4/projects/898/packages/pypi/simple \
    --extra-index-url https://$GITLAB_PYPI_INDEX_USERNAME:$GITLAB_PYPI_INDEX_PASSWORD@gitlab.transzap.com/api/v4/projects/899/packages/pypi/simple \
    dagster-aws-ext dagster-dask-ext dagster-pandas-ext

A hypothetical group-level index should allow me to install all three modules with one index URL.

pip install \
    --extra-index-url https://$GITLAB_PYPI_INDEX_USERNAME:$GITLAB_PYPI_INDEX_PASSWORD@gitlab.transzap.com/api/v4/groups/259/packages/pypi/simple \
    dagster-aws-ext dagster-dask-ext dagster-pandas-ext

Permissions and Security

Access to the modules should still follow the project each module lives in, like how Docker images are handled.

Documentation

Availability & Testing

What does success look like, and how can we measure that?

It should be easy to use modules from multiple projects published in GitLab PyPi package registries. Users should not have to manage multiple indexes.

What is the type of buyer?

GitLab PyPi Repository is available in GitLab Silver/Premium.

Is this a cross-stage feature?

Links / references

https://gitlab.my.salesforce.com/0016100000KvacF

Edited by Steve Abrams