Use Go Package files as a cache for Go proxy queries

!34558 (merged) introduces a way to create a Packages::Package (GitLab Package entity) and Packages::PackageFiles (files associated with a package entity) from a Packages::Go::ModuleVersion (version of a Go module).

Following this, the Go proxy should be modified such that, when the proxy receives a request for a resource of a Go module version, the proxy A) uses the corresponding PackageFile of the corresponding Package if one exists, or B) schedules a worker to create the missing package and its files.

Implementation (diff)

diff --git a/lib/api/go_proxy.rb b/lib/api/go_proxy.rb
index c0207f9169c..2c6eeb90f92 100755
--- a/lib/api/go_proxy.rb
+++ b/lib/api/go_proxy.rb
@@ -64,6 +64,24 @@ module API
       rescue ArgumentError
         not_found!
       end
+
+      def present_file_if_exists(ver, type)
+        package_file = ver.package && Packages::PackageFileFinder.new(ver.package, "#{ver.name}.#{type}").execute
+        if package_file
+          return present_carrierwave_file!(package_file.file)
+        end
+
+        # Allow any user or unauthenticated request to trigger package
+        # synchronization, but gate it behind a lease to prevent excessive job
+        # enqueuing
+        if Gitlab::ExclusiveLease.new("go_proxy:sync_packages:#{ver.full_name}", timeout: 1.hour).try_obtain
+          ::Packages::Go::SyncPackagesService
+            .new(user_project, current_user, ver.name, ver.mod.path)
+            .execute_async
+        end
+
+        yield
+      end
     end
 
     params do
@@ -109,8 +127,10 @@ module API
         get ':module_version.mod', requirements: MODULE_VERSION_REQUIREMENTS do
           ver = find_version
 
-          content_type 'text/plain'
-          ver.gomod
+          present_file_if_exists(ver, :mod) do
+            content_type 'text/plain'
+            ver.gomod
+          end
         end
 
         desc 'Get a zip of the source of the given module version' do
@@ -122,12 +142,14 @@ module API
         get ':module_version.zip', requirements: MODULE_VERSION_REQUIREMENTS do
           ver = find_version
 
-          content_type 'application/zip'
-          env['api.format'] = :binary
-          header['Content-Disposition'] = ActionDispatch::Http::ContentDisposition.format(disposition: 'attachment', filename: ver.name + '.zip')
-          header['Content-Transfer-Encoding'] = 'binary'
-          status :ok
-          body ver.archive.string
+          present_file_if_exists(ver, :zip) do
+            content_type 'application/zip'
+            env['api.format'] = :binary
+            header['Content-Disposition'] = ActionDispatch::Http::ContentDisposition.format(disposition: 'attachment', filename: ver.name + '.zip')
+            header['Content-Transfer-Encoding'] = 'binary'
+            status :ok
+            body ver.archive.string
+          end
         end
       end
     end

Original MR description, selected comments

Per this comment, this MR adds:

• A golang package type for Packages::Package
• A service to create Go Packages and PackageFiles
• A worker to create Go Packages and PackageFiles asynchronously

The Go proxy will then use Packages and PackageFiles as a cache to avoid Gitaly calls when possible, as well as triggering the worker when missing entries are discovered.

Closes #220628 (closed)

@10io @sabrams What do you think about gating this behind user_project.repository_languages? As in, if the request is unauthenticated, don't schedule SyncPackagesService unless the project is known to contain Go files.

The lease will prevent excessive job queuing of a single project. A malicious agent could still enqueue jobs for every single public project on GitLab.com. I assume SideKiq would not allow this to starve GitLab of resources, but it could delay legitimate SyncPackagesService jobs. Preventing unauthenticated requests from enqueueing jobs for invalid projects would not eliminate the potential for abuse but would limit it. And an authenticated user making 10s or 100s of thousands of requests could be banned.