Support S3 server side encryption in CI cloud native job logs
Previously if S3 server side encryption were required to store CI artifacts via a bucket policy, the final migration to object storage would fail, leaving the job log unmigrated.
To fix this, we need to send the right headers (e.g. x-amz-server-side-encryption
) to the object storage backend via Fog.
Relates to #273498 (closed)
Testing
- Set up an S3 bucket with this policy that rejects any uploads that don't have the right server side encryption headers:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*"
},
{
"Sid": "DenyIncorrectEncryptionHeader",
"Effect": "Deny",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "AES256"
}
}
},
{
"Sid": "DenyUnEncryptedObjectUploads",
"Effect": "Deny",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
"Condition": {
"Null": {
"s3:x-amz-server-side-encryption": "true"
}
}
}
]
}
- Enable CI live job traces:
Feature.enable(:ci_enable_live_trace)
- Create a simple CI job:
image: alpine:latest
test:
script:
- date
- echo "hello" > test.txt
- apk add curl
- curl -o guide.txt http://www.textfiles.com/programming/cguide_3.txt
- cat guide.txt
artifacts:
paths:
- test.txt
- Configure GitLab for object storage:
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
'region' => 'us-west-2',
'use_iam_profile' => true
}
gitlab_rails['object_store']['storage_options'] = { 'server_side_encryption' => 'AES256' }
- Run the CI job. Before this merge request, this error would be seen in Sidekiq:
Before
{
"severity": "WARN",
"time": "2020-11-12T21:34:38.282Z",
"error_class": "Excon::Error::Forbidden",
"error_message": "Expected(200) <=> Actual(403 Forbidden)\nexcon.error.response\n :body => \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6246E8900FE3422F</RequestId><HostId>y8nNy7TgayMAcZMTzJZ9B5t3u1lSZiDuvFRYGaOHOMHiaNDJE4U1TCrt23u+DK6fUEKNK7UXk50=</HostId></Error>\"\n :cookies => [\n ]\n :headers => {\n \"Connection\" => \"close\"\n \"Content-Type\" => \"application/xml\"\n \"Date\" => \"Thu, 12 Nov 2020 21:33:56 GMT\"\n \"Server\" => \"AmazonS3\"\n \"x-amz-id-2\" => \"y8nNy7TgayMAcZMTzJZ9B5t3u1lSZiDuvFRYGaOHOMHiaNDJE4U1TCrt23u+DK6fUEKNK7UXk50=\"\n \"x-amz-request-id\" => \"6246E8900FE3422F\"\n }\n :host => \"stanhu-s3-workhorse-testing.s3-us-west-2.amazonaws.com\"\n :local_address => \"172.30.0.215\"\n :local_port => 37564\n :path => \"/tmp/builds/104/chunks/0.log\"\n :port => 443\n :reason_phrase => \"Forbidden\"\n :remote_ip => \"52.218.244.49\"\n :status => 403\n :status_line => \"HTTP/1.1 403 Forbidden\\r\\n\"\n",
"context": "Job raised exception",
"class": "Ci::BuildTraceChunkFlushWorker",
"args": [
"93"
],
"retry": 3,
"queue": "pipeline_background:ci_build_trace_chunk_flush",
"version": 0,
"queue_namespace": "pipeline_background",
"jid": "5839366323d57a8f50f07c02",
"created_at": "2020-11-12T21:33:56.783Z",
"meta.user": "root",
"meta.project": "root/simple-ci2",
"meta.root_namespace": "root",
"meta.caller_id": "/api/:version/jobs/:id/trace",
"meta.feature_category": "continuous_integration",
"correlation_id": "b8871dc7a4df9f803c9cea9fc9708406",
"enqueued_at": "2020-11-12T21:34:37.084Z",
"failed_at": "2020-11-12T21:33:57.131Z",
"retry_count": 0,
"db_count": 2,
"db_write_count": 0,
"db_cached_count": 0,
"error_backtrace": [
"app/models/ci/build_trace_chunks/fog.rb:18:in `set_data'",
"app/models/ci/build_trace_chunk.rb:250:in `unsafe_set_data!'",
...
After:
irb(main):020:0> b = Ci::Build.find(106)
=> #<Ci::Build id: 106, status: "success", finished_at: "2020-11-12 21:41:16", trace: nil, created_at: "2020-11-12 21:41:00", updated_at: "2020-11-1...
irb(main):021:0> b.job_artifacts.last.file.path
=> "85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61/2020_11_12/106/153/job.log"
Edited by Stan Hu