Skip to content

Enable multipart transfers for S3 copying

Stan Hu requested to merge sh-fog-aws-set-multipart-chunk-size into master

This is a follow-up to !50922 (merged). Multithreaded, multipart transfers are only enabled if the file is above 5 GB or the multipart chunk size is explicitly set.

We now set the chunk size to 10 megabytes (the AWS SDK default) to make file copies faster.

This commit also updates fog-aws to v3.8.0 to work properly with server side encryption parameters.

CHANGELOG: https://github.com/fog/fog-aws/blob/master/CHANGELOG.md

Testing

A 3.9 GB artifact copied in 14 seconds as opposed to minutes:

{
  "time": "2021-01-13T23:35:36.956Z",
  "severity": "INFO",
  "duration_s": 14.30236,
  "db_duration_s": 0.01398,
  "view_duration_s": 14.28838,
  "status": 201,
  "method": "POST",
  "path": "/api/v4/jobs/284/artifacts",
  "params": [
    {
      "key": "artifact_format",
      "value": "zip"
    },
    {
      "key": "artifact_type",
      "value": "archive"
    },
    {
      "key": "file.remote_url",
      "value": "https://xxx"
    },
    {
      "key": "file.size",
      "value": "3925920896"
    },
    {
      "key": "file.sha256",
      "value": "81ae90293ab35ba42b3d3b8e219b5faf445c366359178b25f975059628b3e38d"
    },
    {
      "key": "file.md5",
      "value": "8f0d1ca70a8e83eea3c019ac730fa61d"
    },

Closes #216442 (closed)

Edited by Stan Hu

Merge request reports

Loading