Add service to index nuget symbol files
What does this MR do and why?
In !129916 (merged), we created the packages_nuget_symbols
database table. In this MR, we introduce the services needed to create records to be inserted in this table.
In the NuGet Repository, we support publishing symbol packages. But to make use of those packages, they need to be indexed.
The indexing process means extracting the debugging files .pdb
from the uploaded symbol package .snupkg
, and for each extracted file .pdb
we create a corresponding database record in the packages_nuget_symbols
table.
The packages_nuget_symbols
database record holds the following information about the .pdb
file:
- file: the reference to the
.pdb
file on the object store. -
signature: This is a unique GUID and it's embedded in the
.pdb
file. It's extracted from the file byPackages::Nuget::ExtractSymbolSignatureService
. When debuggers such asVisual Studio
need to retrieve the symbol debugging files from symbol servers, they send this signature to the server to retrieve the correct files. - file_path: It's the path of the .pdb file in the directory structure of the symbol package
.snupkg
. For ex: "lib/netstandard2.0/MyPackage.pdb" - size: the size of the
.pdb
file. - package_id: the id of the parent NuGet package in the
packages_packgaes
table.
Now to summarize the flow of pushing a symbol package to the NuGet Repository:
- The user would push a symbol package using nuget.cli:
nuget push package.1.0.0.snupkg -Source gitlab
- When the server receives the package file
package.1.0.0.snupkg
, it processes it as follows:- Extract the metadata
.nuspec
file which contains the necessary metadata such as the name and the version of the package. - Extract the .pdb files and index them; meaning create a database record for each file in the
packages_nuget_symbols
table, and upload the file to the object storage. (this step is introduced in this MR)
- Extract the metadata
Technical implementation:
Before this MR, we had one service responsible for fetching the package file from the object store and extract the metadata .nuspec
file from it. But in this MR, I needed to separate this service into two services: one is responsible for the package file from the object store, and the other is responsible for extracting the metadata .nuspec
file.
This refactor was needed because the newly introduced service Packages::Nuget::CreateSymbolFilesService
needs the fetched package file to extract the .pdb
files from it. That means we need to fetch the package file once, and then use it twice: in the metadata .nuspec
file extraction service, and in the symbol .pdb
files creation service.
Now we can summarize how the services are structured:
-
Packages::Nuget::ProcessPackageFileService
: New service which is responsible for downloading the package file from the object store and then passing it to the other two services; -
Packages::Nuget::ExtractMetadataFileService
: Existing refactored service that receives the downloaded package zip file and extracts the.nuspec
metadata file. -
Packages::Nuget::CreateSymbolFilesService
: New service that receives the downloaded package zip file and indexes the symbol.pdb
files.
I wanted the symbols creation service to be as quiet as possible. It doesn't complain or raise exceptions. It just tries to create database records and upload files for the extracted .pdb
files. If there's a uniqueness violation or unsuccessful signature extraction, the new record will silently fail and be skipped. I thought this could be beneficial in the beginning since we don't want to interrupt the main package uploading process with any raised exceptions from the newly introduced indexing service. Once it is deemed stable and functional, we can enhance the flow as needed.
The indexing service is behind the index_nuget_symbol_files
feature flag to gradually roll out the feature and revert easily in case of any errors.
Packages::Nuget::ExtractSymbolSignatureService
:
Implementation of To implement the logic of extracting the signature from the .pdb
files, I used this page as a reference to know about the structure of the signature and how it's constructed. However, the implementation of the service is done after several trial and error attempts. Therefore, it's not a well-known implementation because of the lack of such a thing. I tried to look for a ready-to-use solution on how to extract this piece of information from .pdb
files with no luck.
Known limitation:
The implementation targets only the Microsoft .Net portable PDB format files. Other symbol files will not be indexed correctly.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
- Make sure you have the NuGet CLI installed (see nuget docs for links to installation pages).
- Add a project as your NuGet source in your local GitLab NuGet Repository:
nuget source Add -Name localhost -Source "http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json" -UserName <gitlab_username> -Password <personal_access_token>
- In the rails console, enable the feature flag for the project:
Feature.enable(:index_nuget_symbol_files, Project.find(<project_id>))
- Push a NuGet package that has a symbol to your project. You can test using this one. Download both the
.nupkg
&.snupkg
files in the same directory, and then push the.nupkg
. The.snupkg
file should be pushed automatically after the.nupkg
is pushed.
nuget push Package.nupkg -Source localhost
- Once the package is published, in the rails console verify that the symbol records have been created:
Packages::Nuget::Symbol.where(package_id: <package_id>)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #416177 (closed)