Fix a dependency proxy upload premature interruption in workhorse

🔭 Context

The dependencyproxy package of workhorse is a pretty smart feature.

As its core, it flows like this:

1. A client asks for a resource.
2. Rails doesn't have that resource but an upstream server can have it. It returns the dependencyproxy "instruction" to workhorse.
3. When receiving this "instruction", workhorse will:
4. Get the file from the upstream url.
5. Stream it back to the client.
6. At the same time, upload it to rails.

You might wonder how 3.2 and 3.3 is done? Well, simply with a io.TeeReader. More details in this blog post.

In very short words, we are 🤹 with several requests and flowing a response into a request and a response.

This was implemented with simple http.NewRequestWithContext and we used the context of the client connection (from 1.) to bind the contexts of these other requests (the request to the upstream resource and the upload request).

With the release of the Maven dependency proxy, which heavily use the above logic, we noticed that we have a small flaw: the client could close its connection because it received everything that was required before the upload logic (3.3) was done. This would cancel the upload request. Notice that the client (from 1.) would still receive the proper file from the upstream. The only thing that doesn't work here is the upload, everything else would work just fine.

This has been described in issue Dependency proxy: some files will interrupt the... (#448886 - closed). We didn't dig too much why the client closes its connection early but it seems to depend on the size of the file fetched from the upstream. We are assuming that writing back the client (1.) might use a buffer and in some conditions, a buffer is flushed to the client and it happens that it's exactly the last bytes of the file content: after receiving that chunk, the client is all done and closes the connection.

🤔 What does this MR do and why?

• In the workhorse dependency proxy, for the upload request, don't re-use the context of the request of the external client.
• Add a related spec that simulates a client that closes early a connection.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

I have some doubts on using a context.Background() context that will need to be cleared.

🦄 Screenshots or screen recordings

No UI changes

⚗ How to set up and validate locally

We're going to set up a maven dependency proxy and use $ curl to simulate the external client on specific files.

For the files, we are going to use this snippet: https://gitlab.com/-/snippets/3686132. It contains two files: one that will make the dependency proxy request succeed and one that it will make it fail.

1. Create a project and get a PAT with the api scope.
2. In the Packages and Registries settings, enable the maven dependency proxy and use the following url: https://gitlab.com/-/snippets/3686132/raw.
   • A license is required for this as the dependency proxy is a premium+ feature.

1️⃣ With master

Let's pull the ok file:

$ curl "http://<username>:<PAT>@gdk.test:8000/api/v4/projects/<project_id>/dependency_proxy/packages/maven/main/foo/bar/ok.xml"
<?xml version="1.0" encoding="UTF-8"?>
[snip .. snip]

• Our $ curl command receives the file.
• In the project's package registry, a package has been created.

Delete the package in the package registry. Let's pull the ko file.

$ curl "http://<username>:<PAT>@gdk.test:8000/api/v4/projects/<project_id>/dependency_proxy/packages/maven/main/foo/bar/ko.xml"
<?xml version="1.0" encoding="UTF-8"?>

• Our $ curl command receives the file. As we said above, the client still get the correct file.
• In the project's package registry, there is no package created.

In addition, the workhorse log will show:

2024-04-18_12:28:50.56154 gitlab-workhorse      : {"correlation_id":"01HVRJWP5S53W57T7FFBX1NCRT","duration_ms":0,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"PUT","msg":"","time":"2024-04-18T14:28:50+02:00","uri":""}
2024-04-18_12:28:50.56178 gitlab-workhorse      : {"code":499,"correlation_id":"01HVRJWP5S53W57T7FFBX1NCRT","error":"dependency proxy: failed to upload file","level":"error","method":"GET","msg":"","time":"2024-04-18T14:28:50+02:00","uri":"/api/v4/projects/309/dependency_proxy/packages/maven/main/foo/bar/ko.xml"}
2024-04-18_12:28:50.56184 gitlab-workhorse      : {"content_type":"text/plain; charset=utf-8","correlation_id":"01HVRJWP5S53W57T7FFBX1NCRT","duration_ms":328,"host":"gdk.test:8000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.16.123.1:55890","remote_ip":"172.16.123.1","route":"^/api/","status":200,"system":"http","time":"2024-04-18T14:28:50+02:00","ttfb_ms":327,"uri":"/api/v4/projects/309/dependency_proxy/packages/maven/main/foo/bar/ko.xml","user_agent":"curl/8.1.2","written_bytes":6254}

Showing that the overall request (3rd line) was successful (status 200) but the upload request (1st and 2nd lines) failed due to a context canceled.

2️⃣ With this MR

In the project, browse the package registry and remove all packages that were uploaded.

Let's pull the ok file:

$ curl "http://<username>:<PAT>@gdk.test:8000/api/v4/projects/<project_id>/dependency_proxy/packages/maven/main/foo/bar/ok.xml"
<?xml version="1.0" encoding="UTF-8"?>
[snip .. snip]

• Our $ curl command receives the file.
• In the project's package registry, a package has been created.

Delete the package in the package registry. Let's pull the ko file.

$ curl "http://<username>:<PAT>@gdk.test:8000/api/v4/projects/<project_id>/dependency_proxy/packages/maven/main/foo/bar/ko.xml"
<?xml version="1.0" encoding="UTF-8"?>

• Our $ curl command receives the file.
• In the project's package registry, a package has been created. 🎉

Check the workhorse logs:

2024-04-18_12:35:39.02784 gitlab-workhorse      : {"content_type":"text/plain; charset=utf-8","correlation_id":"01HVRK94MYG3RNYRAAA4FTQDX7","duration_ms":756,"host":"gdk.test:8000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.16.123.1:55947","remote_ip":"172.16.123.1","route":"^/api/","status":200,"system":"http","time":"2024-04-18T14:35:39+02:00","ttfb_ms":621,"uri":"/api/v4/projects/309/dependency_proxy/packages/maven/main/foo/bar/ko.xml","user_agent":"curl/8.1.2","written_bytes":6254}

No errors on the upload 🎉