Implement Dependency proxy via Workhorse injectors
The Dependency Proxy allows users to reduce risk and reliance on external dependencies by caching frequently used packages for fast, reliable access. However, since the Rails app downloads the blobs from upstream sources, it's possible that while downloading is in progress, it will block other requests.
In order to fix the issue, we plan to move the implementation to Workhorse.
sequenceDiagram
Client->>Workhorse: GET /v2/*group_id/dependency_proxy/containers/*image/blobs/:sha
Workhorse->>Rails: GET /v2/*group_id/dependency_proxy/containers/*image/blobs/:sha
Rails->>Rails: Check DB. Is blob persisted in cache?
alt In Cache
Rails->>Workhorse: Respond with send-url injector
Workhorse->>Client: Send the file to the client
else Not In Cache
Rails->>Rails: Generate auth token and download URL for the blob in upstream registry
Rails->>Workhorse: Respond with send-dependency injector
Workhorse->>Container Registry: Open stream for a blob
Container Registry->>Workhorse: Stream
Workhorse->>Rails: GET /v2/*group_id/dependency_proxy/containers/*image/blobs/:sha/authorize
Rails->>Workhorse: Respond with upload instructions
Workhorse->>Client: Send the file to the client
Workhorse->>Object Storage: Save the file
Workhorse->>Rails: Finalize the upload
end
Each of the plain approaches (injectors or pre-auth) has a flaw:
- For the plain injectors approach !68157 (merged), it's not trivial to finalize the file upload (retrieving the upload options and performing a request to Rails). We could send authorize options as injector params, but it would result in code duplication with BodyUploader
- For pre-auth approach !68159 (closed), it's not trivial to send the cached file.
send-url
injector doesn't work without modifications, sending file url and treating it as a just another remote url would result in code dupliation with send-url injector
The idea is to combine both approaches:
- When blob is requested:
- If it exists in the database, it's sent via Workhorse (
send_upload
) - If it doesn't exist, Workhorse downloads it from the remote and streams the file to the user; after that, it uploads the file via Workhorse using a pre-authorized request
- If it exists in the database, it's sent via Workhorse (
Related issue: #335563 (closed)
Edited by Igor Drozdov