Add an ENTRYPOINT script to the GitLab Runner helper image

Initial issue description: Unable to upload cache to minio S3 server due to x509 unknown certificate authority

I'm running a self-hosted Minio S3 storage server, with a certificate signed by a private authority. Gitlab Runner is running the Kubernetes executor, and has the following cache settings:

[runners.cache]
    Type = "s3"
    ServerAddress = "minio.myserver.com"
    AccessKey = "minio-access-key"
    SecretKey = "minio-secret-key"
    BucketName = "gitlab-ci-cache"

The tests run fine, however at checking and updating cache, I get the following error:

FATAL: Get https://minio.myserver.com/gitlab-ci-cache/runner/b77179e5/project/6/master?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZTEPIPIRNBNNCYMSUCYV%2F20180618%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180618T145218Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b78a9f63ba05cec3cf6655a50e27549ea0ff6816db50f83ba1316c9c2ce03378: x509: certificate signed by unknown authority 

Which would be understandable as it is a private certificate authority. However, I have added said CA to the runner's alpine image, both as described in the documentation, and also with update-ca-certificates. That does work for reaching the Gitlab server itself as described, the cache still doesn't work though.

I've also added it to the test image (both docker:stable, and docker:stable-dind, as I'm running a docker-in-docker configuration) to no avail.

At this point I'm not entirely sure which container communicates with the S3 server, as all of them have the certificate authority installed, yet the error persists.

Smudge error: Error downloading...x509: certificate signed by unknown authority with LFS remote object store

When using remote object store for LFS, the runner gets Smudge error: Error downloading...x509: certificate signed by unknown authority when trying to download objects.

When the first request - request for a new job - is done, Runner is:

• getting the TLS certificate and chain from the GitLab's response,
• trying to generate a full certificates chain based on the response data,
• attaching resolved chain to the job data,
• attaching specified client key/certificate pair (if configured) to the job data.

The chain from the job data is next exported as GIT_SSL_CAINFO variable during job execution. This becomes the only source of truth for git's certificate validation. If the client key/certificate pair was configured, their content is also exported as GIT_SSL_KEY and GIT_SSL_CERT variables which allows git to handle TLS Client Authorization that may be configured in front of GitLab.

The problem here is that the GIT_SSL_CAINFO contains a CA chain that is able to verify only the certificate used by the GitLab instance, from where the job was requested. We've already found this to be problematic for cases when external submodules available through HTTPs are used.

Here we have a similar problem caused by LFS. LFS operation is handled by git. Git connects with GitLab (so the certificate at this point is verified properly), but GitLab sends a redirection to S3. Git tries next to connect to S3, but it still is limited by GIT_SSL_CERT. Unless the S3 endpoint is signed by a certificate verified by the same CA, this will fail at certificate verification stage. And since they are using Amazon S3, then the domain (and certificate) are *.s3.amazonaws.com. While GitLab's domain is example gitlab.example.com. And this can't be verified.

Workaround

Update the helper image to have the ca chain for the self-signed certificate trusted.

FROM gitlab/gitlab-runner-helper:x86_64-latest

RUN apk add --no-cache ca-certificates

COPY ca.crt /usr/local/share/ca-certificates/ca.crt

RUN update-ca-certificates

RUN rm /usr/local/share/ca-certificates/ca.crt

docker build -t registry.gitlab.com/namespace/project/tools/gitlab-runner-helper:$SOME_TAG

Override the helper image used by GitLab by updating the config.toml to use the image you just build with the correct CA trusted.

If you are using the helm chart you can define KUBERNETES_HELPER_CPU_LIMIT environment variable and define it in envVars

Proposal

NOTE: Mounting a directory and updating the certificates will not work because the helper image is used to create containers multiple types (once per stage).

1. Add an entrypoint to the helper image which will look for a certificate file in well-known location /etc/gitlab-runner/certs/ca.crt and install it at every stage.
2. Make sure the entrypoint works as expected.
3. Add a section in the documentation documenting how to leverage the entrypoint to add a certificate to the helper container.

Benefits:

• This features enables users to leverage a custom CA root certificate in a build script (e.g., to do a secondary Git clone or fetch a file through HTTPS).
• This would give 100% control to the user on how certificates should be added to the system since they know their system best.
• Users wouldn't have to wait on GitLab to update our integration to add the certificate to the pool.