Put back required authentication in NuGet group-level endpoints

Context

In Allow anyone to pull public NuGet packages on g... (!155119 - merged), we tried to enable the feature that allow anyone to download NuGet packages from public registries, even if the parent group is private.

To do so, we introduced a new permission read_package_within_public_registries which will be granted if the group has any public registries, regardless of its access level. However, this broke how NuGet handles the requests for private registries.

Mainly, NuGet client sends an anonymous request to the registry (GitLab's NuGet registry in this case), and if the registry responded with 401, NuGet attaches the token to the headers of the next request, so the request can be authenticated & authorized.

After enabling the allow_anyone_to_pull_public_nuget_packages_on_group_level feature flag, the needed 401 response wasn't sent in case if the group has any public registry. In this case, the new permission read_package_within_public_registries (introduced in !155119 (merged)) would be granted, and the anonymous request will pass the authorization phase and enter the finder.

The finder will look for the requested package that its project is public, or its registry is public. But if the requested package is in a private project/registry, the finder will return an empty result, and a 404 response will be returned to NuGet client. Which means NuGet will not be able to send the needed credentials in a subsequent request since it didn't receive the needed 401 from the NuGet registry.

What does this MR do and why?

Put back the required authentication before accessing any NuGet Repository group-level endpoints. This would ensure we always respond with 401 if we received an anonymous request from NuGet clients.

However, this would put a limitation on allow anyone to pull from public package registries feature. anyone here would mean any GitLab user.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

1. Make sure you have 2 private projects in a private group.

2. Open rails console:

   # Enable the ~"feature flag"
   Feature.enable(:allow_anyone_to_pull_public_nuget_packages_on_group_level)
   
   # Enable `package_registry_allow_anyone_to_pull_option` application setting
   ApplicationSetting.last.update(package_registry_allow_anyone_to_pull_option: true)
   
   # For one of the 2 private projects, enable Allow anyone to pull from Package Registry
   Project.find(<project1_id>).project_feature.update(package_registry_access_level: ::ProjectFeature::PUBLIC)
   
   # stub file upload
   def fixture_file_upload(*args, **kwargs)
     Rack::Test::UploadedFile.new(*args, **kwargs)
   end
   
   # Create a nuget package in the other private project (not the one with the public package registry)
   package = FactoryBot.create(:nuget_package, project_id: <project2_id>, package_name: 'hello.nuget')

3. We can now try installing the package using NuGet CLI:

   nuget install hello.nuget -OutputDirectory <output_directory> -Source "http://gdk.test:3000/api/v4/groups/<group_id>/-/packages/nuget/index.json"

   nuget install will show a prompt asking for username and password, which means we correctly respond with 401 and that's why NuGet client tries to get the credentials to send them back with a subsequent request.

On master

• Clear NuGet cache and repeat step 3. ➡ 404 Not found (wrong response)

Related to #471326 (closed)