Skip to content

Support AWS S3 multipart uploads via scoped temporary credentials

Stan Hu requested to merge sh-aws-multipart-with-rolearn into main

What does this MR do?

Previously we tried to support multipart uploads by passing a GoCloud URL and bucket-wide S3 credentials to the cache uploader, but this opens a significant security risk. Because the runner manager and builds can execute on different machines, it is easy to leak these credentials on any given build, giving an attacker full access to the cache. Pre-signed URLs avoid this problem by allowing the cache uploader temporary access to write one specific blob.

However, pre-signed S3 PutObject requests are limited to 5 GB. Larger files require using the multipart API.

Instead, we can use Amazon Security Token Service (STS) to create temporary credentials with a restrictive access scope: the credentials can only be used to upload a specific bucket and object, just as presigned URLs do. This is a similar approach to Azure's shared access signature (SAS).

This means that if users want multipart uploads, they will have to go through an extra step of configuring a role via the UploadRoleARN parameter that can create these narrowly scoped credentials. Other downsides:

  • This only works with AWS.
  • The S3 access credentials already configured must have AssumeRole permission for the given ARN.

Relates to #26921 (closed)

Why was this MR needed?

What's the best way to test this MR?

What are the relevant issue numbers?

Edited by Stan Hu

Merge request reports

Loading