Skip to content

Publish nuget packages on the fly

Moaz Khalifa requested to merge 506035-create-nuget-packages-on-the-fly into master

Context

In the NuGet Repository, we publish new packages using a background worker. The reason is that we don't receive the package name and version for the uploaded package file, but we receive just an archive with the generic name package.nupkg or package.snupkg. That means we need to unzip the uploaded package file to extract the name and version from the metadata file .nuspec.

This unzipping process could be expensive, and the uploaded files could be large. That's why we use a background worker to do all this heavy lifting.

In Add service to handle disallowing duplicate NuG... (!128269 - merged), we implemented a service that could stream the uploaded package file in chunks and extract the metadata file. We assume that the metadata file always is within the first few chunks of the archive, so we download a max of 5 chunks, and in most uploads, this is enough to get the metadata file .nuspec.

This service was used only to handle allowing/disallowing the uploaded packages' duplication. However, we wanted to expand its usage so that we can upload the packages directly without the need for the background worker. Because this service would give us the uploaded package's name & version, and using this information, we can create the package on the fly.

There are a couple of benefits to this new behavior:

  • Avoid creating a temp package before handling the upload in the background worker. This temp package can be destroyed later, so this would save wasting the database table PKs.
  • Returning a final response to the user. With the background worker, we return 201 without being sure that the upload will succeed. Later, if the upload failed for any reason, we display the error in the UI. This could be misleading sometimes and could affect the flow of publication.
  • Mitigate NuGet package registry: investigate sequential ... (#415552). With the background worker, we can get a situation of parallel upload between the main package and its symbol, which could result in rejecting a legit symbol package upload.
  • Decrease Sidekiq load and avoid downloading the whole package archive in the background worker. Instead, we just download a couple of chunks and that's it.

However, we also have some limitations:

  • We still need to handle the symbol packages .snupkg upload in the background worker. That's because we have to extract the debugging symbol files .pdb from the symbol package archive, and to do that, we have to download the whole archive and unzip it. We cannot do that during the web request.
  • For some packages, we might not be able to find the metadata file .nuspec in the first couple of chunks, so in this case, we still need to fall back to the background worker. That means we now check if the .nuspec file extraction is successful, we proceed with the package creation on the fly. Otherwise, we create a temp package and ask the background worker to handle the upload.

What does this MR do and why?

  • Add a new conditional branch in the API class API::NugetProjectPackages to check the ability of extracting the metadata file from the uploaded package archive. If it's successful, create the package and its associations (metadata, tags, dependencies, build infos...etc.) on the fly. Otherwise, fallback to the background worker.
  • Create a new service Packages::Nuget::CreateOrUpdatePackageService that's responsible for creating or updating the package on the fly.
  • Make a couple of needed changes to some services responses so that services can better communicate with each other.
  • Gate the new path behind a feature flag as an additional safety net.
  • Add covering specs.

References

Please include cross-links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

1st case: A package that we can extract its metadata file on the fly

  1. Make sure that the Object Storage is enabled in your GDK.
  2. Enable the feature flag: 
    Feature.enable(:create_nuget_packages_on_the_fly)
  3. Download this package to test with.
  4. Tail into the logs to observe which code path is executed: 
    tail -f gitlab/log/development.log | grep -E 'packages/nuget/create_or_update_package_service|packages/nuget/extraction_worker'
  5. In the directory where the package archive .nupkg was downloaded, open the terminal and publish it to a GitLab project: 
    dotnet nuget push newtonsoft.json.13.0.3.nupkg --source http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json --api-key <PAT>
  6. Take a look at the logs, and you should see that the package was created inside the Packages::Nuget::CreateOrUpdatePackageService, which means it's created on the fly, not in the background worker.
  7. Disable the feature flag and destroy the created package, then republish it and look at the logs. We should see some logs for the Packages::Nuget::ExtractionWorker, which means it's created by the background worker.

2nd case: A package that we cannot extract its metadata file on the fly

  1. Download this package to test with.
  2. Make sure the feature flag is enabled.
  3. Publish the package: 
    dotnet nuget push visioforge.dotnet.core.redist.videocapture.x86.2025.0.3.nupkg --source http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json --api-key <PAT>
  4. In the logs, we should see the Packages::Nuget::ExtractionWorker was enqueued, which means that we weren't able to extract the metadata file on the fly, and fell back to the worker to handle the upload.
  5. Disable the worker, and destroy the package and republish it. The worker will handle the upload as well.

3rd case: Uploading a symbol package

Follow the steps 1 & 2 from here when the feature flag is enabled. The main package .nupkg should be published on the fly, while the symbol package .snupkg should be published by the background worker.

Related to #506035

Edited by Moaz Khalifa

Merge request reports