Support AWS SSE-KMS in backups
What does this MR do?
As described in https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html, AWS supports three different modes for encrypting S3 data:
- Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)
- Server-Side Encryption with Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS)
- Server-Side Encryption with Customer-Provided Keys (SSE-C)
Previously, SSE-S3 and SSE-C were supported via the
backup.upload.encryption
and backup.upload.encryption_key
configuration options.
SSE-KMS was previously not supported in backups because there was no way to specify which customer-managed key to use. However, we did support SSE-KMS with consolidated object storage enabled for other CI artifacts, attachments, LFS, etc. Note that SSE-C is NOT supported here.
In consolidated object storage, the storage_options
Hash provides the
server_side_encryption
and server_side_encryption_kms_key_id
parameters that allow admins to configure SSE-KMS. We reuse this
configuration in backups to support SSE-KMS.
Relates to #338764 (closed), #328763
Testing
- Configure
gitlab.yml
with AWS credentials:
backup:
path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/)
gitaly_backup_path: /Users/stanhu/gitlab/gdk-ee/gitaly/_build/bin/gitaly-backup # Path of the gitaly-backup binary (default: searches $PATH)
# archive_permissions: 0640 # Permissions for the resulting backup.tar file (default: 0600)
# keep_time: 604800 # default: 0 (forever) (in seconds)
# pg_schema: public # default: nil, it means that all schemas will be backed up
upload:
remote_directory: stanhu-s3-workhorse-testing
connection:
provider: AWS
region: us-west-2
aws_access_key_id: REDACTED
aws_secret_access_key: REDACTED
storage_options:
server_side_encryption: 'aws:kms'
server_side_encryption_kms_key_id: 'arn:aws:kms:us-west-2:REDACTED'
- Define an S3 bucket policy that rejects uploads that are missing server side encryption headers:
{
"Version": "2012-10-17",
"Id": "PutObjPolicy",
"Statement": [
{
"Sid": "DenyIncorrectEncryptionHeader",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "aws:kms"
}
}
},
{
"Sid": "DenyUnEncryptedObjectUploads",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
"Condition": {
"Null": {
"s3:x-amz-server-side-encryption": "true"
}
}
}
]
}
- Run the backup:
bundle exec rake gitlab:backup:create SKIP=uploads,builds,artifacts,lfs,registry,pages,repositories
- Then check the S3 bucket for the uploaded file: