Add worker to upload npm packages async
What does this MR do and why?
Problem
Due to the upload of NPM packages happening synchronously we face some limitations and aren't able to parse the package tarball.
This leads to missing features like the support for overrides
Additionally, the async upload will bring to us a few other benefits #417975 (comment 1601484030)
Solution
Change the NPM package upload from synchronous to asynchronous.
Implementation
In this MR, we introduce a new background worker that should be responsible for handling the async upload flow.
To move from sync to async upload, we will mark the status
of the uploaded package as processing
. This means the package will appear in the UI, but it's not ready to be used yet till its status
changes from processing
to default
.
The status
update process will happen in the background job.
This worker will fetch the package file from the database using its id
, and then pass the file to a service. The service is going to do two things:
- download the file from the Object Storage and extract the
package.json
from it. - update the package's
status
fromprocessing
todefault
once thepackage.json
is found.
This is a simplified flow and should be followed by more MRs to parse the package.json
as intended.
The changes are gated behind a feature flag since this shift in the upload flow is a breaking change and should be released in milestone 17
.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Enable the feature flag in rails console:
::Feature.enable(:upload_npm_packages_async)
- Publish an NPM package. I used https://gitlab.com/10io/gl_pru#npm.
- The package should be published successfully with
deafult
status.
Related to #433009 (closed)