Respond with 401 if no packages found in groups with public NuGet registries

Context

In Allow anyone to pull public NuGet packages on g... (!155119 - merged), we tried to enable the feature that allow anyone to download NuGet packages from public registries, even if the parent group is private.

To do so, we introduced a new permission; that permission will be granted if the group has any public registries, regardless of its access level. However, this broke how NuGet handles the requests for private registries.

Mainly, NuGet client sends an anonymous request to the registry (GitLab's NuGet registry in this case), and if the registry responded with 401, NuGet attaches the token to the headers of the next request, so the request can be authenticated & authorized.

After enabling the allow_anyone_to_pull_public_nuget_packages_on_group_level feature flag, the needed 401 response wasn't sent in case if the group has any public registry. In this case, the new permission read_package_within_public_registries (introduced in !155119 (merged)) would be granted, and the anonymous request will pass the authorization phase and enter the finder.

The finder will look for the requested package that its project is public, or the registry is public, but if the requested package is in a private project/registry, the finder will return an empty result, and a 404 response will be returned to NuGet client. Which means NuGet will not be able to send the needed credentials since it didn't receive the needed 401 from the NuGet registry.

Solution

We need a way to respond with 401 if the requested package is in a non-public project/registry, even if the group has other public registries.