Skip to content

Add sha256 hash to NuGet symbols

Moaz Khalifa requested to merge 428847-add-sha256-hash-to-nuget-symbols into master

What does this MR do and why?

In the efforts to support the symbol server capability for the NuGet Repository symbol packages, we are now ready to serve the debugging files .pdb to the consuming debuggers such as Visual Studio. How do those debuggers consume the symbol debugging files?

Aside from all the debugger configuration details, what does matter here is the request the debugger sends to the symbol server. The request doesn't have authentication credentials, however, it includes the needed parameters that we can use to make sure the request is legitimate.

The debugger request has this form: <symbol_server_url>/:file_name/:signature/:file_name

By using the :file_name and :signature params, we can serve the right .pdb file. The signature can be considered as a kind of authentication token since it cannot be known unless the debugger has the executable of the Nuget package .dll. The signature is hashed inside this executable .dll and in the debugging file .pdb.

In NuGet Repository, we index the .pdb files by storing their names and signatures. So we can receive the request from the debugger, and then look into the packages_nuget_symbols table to find the matching record using the :file_name and :signature.

However, this is not good enough for security purposes. The .pdb file might have the matching filename and signature, but what if the file itself was tampered with or changed in any harmful way? In this case, the debugger will download a malicious file and we will serve it as well. A lose-lose situation.

To mitigate this threat and make sure the debugger pulls the intended correct file, the request coming from the debugger (Visual Studio for example) usually includes a header named Symbolchecksum. This header holds the sha256 hash of the requested file. This hash is stored in the executable .dll. So the server is responsible for matching this sha256 hash with the requested .pdb file.

Request headers sample: 
{"Host"=>"gdk.test:3000",
 "Gitlab-Workhorse"=>"11-10-0cfa69752d8-74ffd66ae-ee-234461-g93a49ed8ee21",
 "Gitlab-Workhorse-Proxy-Start"=>"1697667453093503000",
 "Symbolchecksum"=>
  "SHA256:3c9ecdd0948114da9579d3edf0dc216b58215fd9f31577f7bbfd664524efb338, SHA256:ba156c4dcfa83a873613e2b4a3cfdcd9115ed7883b862f5678b43ab48764959a, SHA256:2ca70b9ae2e5145d422f53f5dc1a97b2a9aea8de12c586aef0f66486d344aa18, SHA256:fd50dbac10193b8a0449b6ae1ec22a26a55780fa946e8e76ac1f2931ee39dfad",
 "X-Forwarded-For"=>"172.16.123.1",
 "X-Request-Id"=>"01HD2DV4558FQCQDNM4F1VTJFY",
 "X-Sendfile-Type"=>"X-Sendfile",
 "Accept-Encoding"=>"gzip",
 "Version"=>"HTTP/1.1"}

So in this MR, we do two things:

  1. Calculate the sha256 hash for the .pdb files we index and store. More details on how it's calculated: https://github.com/dotnet/corefx/blob/master/src/System.Reflection.Metadata/specs/PE-COFF.md#portable-pdb-checksum
  2. Add a new binary database column named file_sha256 to the packages_nuget_symbols table and save the calculated hash in it.

That would enable us to inspect the coming Symbolchecksum header and match it with what we have, if it matches, then serve the file, otherwise, we return a 404 status code.

Screenshots or screen recordings

N/A

How to set up and validate locally

Follow the same steps mentioned here. When validating the created symbol files in step 5, the field file_sha256 should be correctly populated with the sha256 hash.

How to validate if our computed sha256 hash is the same as the one in the executable?

  1. We need to test with an actual open-source package from nuget.org. We can use the Autofac package.
  2. Download the package .nupkg file and the symbol .snupkg file
  3. Push Autofac package and its symbol to your local gdk NuGet Repository following the same steps mentioned here
  4. In rails console, validate that the Packages::Nuget::Symbol records have been created:
Packages::Package.nuget.last.nuget_symbols
  1. Each Packages::Nuget::Symbol record has file_path & file_sha256 fields. We need those to validate the correctness of our computed sha256 hash.
  2. Unzip the .nupkg file you downloaded for the Autofac package.
  3. In the extracted folder, you should find the corresponding executable for each Packages::Nuget::Symbol record in the same file_path, with one difference: instead of lib/net6.0/Autofac.pdb, the executable has the .dll extension. So the path for the corresponding executable would be lib/net6.0/Autofac.dll.
  4. In the same directory of the executable you chose to test, open an IRB session.
  5. In the IRB, execute the following commands to extract the sha256 from the executable:
content = File.read 'Autofac.dll'
index = content.index('SHA256')
content[(index + 7)..(index + 38)].unpack('H*').last
  1. The return of the last line should be the sha256 hash embedded in the executable. Compare it with file_sha256 field for the corresponding Packages::Nuget::Symbol record and they should be the same.
  2. You can test with any different package (as long as it has a symbol package .snupkg) by repeating the same steps.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #428847 (closed)

Edited by Moaz Khalifa

Merge request reports