Enable multipart transfers for S3 copying
This is a follow-up to !50922 (merged). Multithreaded, multipart transfers are only enabled if the file is above 5 GB or the multipart chunk size is explicitly set.
We now set the chunk size to 10 megabytes (the AWS SDK default) to make file copies faster.
This commit also updates fog-aws to v3.8.0 to work properly with server side encryption parameters.
CHANGELOG: https://github.com/fog/fog-aws/blob/master/CHANGELOG.md
Testing
A 3.9 GB artifact copied in 14 seconds as opposed to minutes:
{
"time": "2021-01-13T23:35:36.956Z",
"severity": "INFO",
"duration_s": 14.30236,
"db_duration_s": 0.01398,
"view_duration_s": 14.28838,
"status": 201,
"method": "POST",
"path": "/api/v4/jobs/284/artifacts",
"params": [
{
"key": "artifact_format",
"value": "zip"
},
{
"key": "artifact_type",
"value": "archive"
},
{
"key": "file.remote_url",
"value": "https://xxx"
},
{
"key": "file.size",
"value": "3925920896"
},
{
"key": "file.sha256",
"value": "81ae90293ab35ba42b3d3b8e219b5faf445c366359178b25f975059628b3e38d"
},
{
"key": "file.md5",
"value": "8f0d1ca70a8e83eea3c019ac730fa61d"
},
Closes #216442 (closed)
Edited by Stan Hu