Skip to content

GitLab reads from Vault for CI Variables

Problem to solve

Secrets stored in HashiCorp Vault are currently not easily read by GitLab for CI Jobs. We want to support a user that has secret stored in Vault and have it used by GitLab.

Intended users

This feature will be used by automation developers who need to access secrets as part of their build and deploy process, providing a new and more secure way to do so.

Proposal

We are supporting the leanest approach for GitLab read a secret from Vault and bring that into a CI Job, using the JWT as an authentication method. The secret would be stored in Vault and retrieved from Vault via the yaml.

We'll add support for the following syntax to .gitlab-ci.yml in order to define Vault servers and secrets from the syntax in #218746 (closed). We will also be delivering the Runner configuration as described in #212252.

secrets:
  # production/s3 is a `path:`, where `secret-key` is a fieldname
  # @path/to/engine indicates a path used by `kv-v2` which we consider to be a default engine
  VAULT_SIMPLE_SECRET:
    vault: production/s3/secret-key

  VAULT_SIMPLE2_SECRET:
    vault: production/s3/secret-key@path/to/engine

  VAULT_COMPLEX_SECRET:
    vault:
      engine:
        name: kv-v2
        path: path/to/engine
      path: production/s3
      field: secret-key

GitLab will parse this section and provide the data to the Runner as part of the job request response. Runner will then authenticate with Vault, fetch the secrets and save them under a path that will be stored in the specified ENV variable (similar to how "File" type CI variables work).

In the first iteration the Vault server definition (url and auth method path) will be defined with CI variables, later we'll add UI for this with #218677 (closed).

Other considerations

  • Start with YAML --> in the future move this into a group or project settings like the Variables framework - will always need to have a section in the YAML
  • Once we have the section in the job YAML, we should continue to expand the YAML

Permissions and Security

Permissions are configured at Vault by using policies, roles, and when configuring the auth method. This should be documented with example, similar to https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault.

Documentation

  • This will need documentation on how this works

Testing

  • Most of the secret management will be contained on the Vault side. For example, secrets expiration and storage will be handled on the Vault Side

_ We'll need vault instance accessible over the internet in order to test this

Links / references

Edited by Krasimir Angelov