Skip to content

Allow user to specify any runner configuration in the runner helm chart

Problem

At the moment we have a disjointed user experience in what a user is able to configure with the runner config.toml and what we allow the user to configure with the values.yml. Every time we add a new configuration field in the Runner, we have to add support for that field in the helm chart, which increases the engineering workload.

Apart from the extra work the maintainers have to do in order to support configuration in the helm chart, it's not always possible to use environment variables/flags to set the registration such as #83 (closed). This is why we created the configuration template so that we can specify a configuration template.

Proposals

Keep the values.yml as it is and just direct the user to specify their own configuration template. This would mean that for anything defined in the values.yml would have to rely on flags/environment variables.

What we can do is to clean this up incrementally:

  1. Introduce the configuration template, freezing anything new under the runners, something similar to what we did in 717506c7
  2. Direct all users to use the configuration template, by updating the docs
  3. Deprecate the runners, if it's possible print a warning message to the user if they are still specifying these values.
  4. In GitLab Runner chart 1.0 we remove runners and have the user completely rely on the configuration template.

So to give an idea of how the values.yml would look like it would be something like:

runners:
  config:
    template: |
    [[runners]]
      output_limit = 1026
      [runners.cache]
        Type = "s3"
        Path = "path/to/prefix"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "s3.amazonaws.com"
          AccessKey = "AWS_S3_ACCESS_KEY"
          SecretKey = "AWS_S3_SECRET_KEY"
          BucketName = "runners-cache"
          BucketLocation = "eu-west-1"
          Insecure = false

Pros

Assuming that we merge !197 (closed) and reject !153 (closed)

  1. There is no work for the maintainers where they add a configuration to the gitlab-runner project and would have to spend time translating it to the helm chart.
  2. It gives full control to what the user wants.
  3. If we add new configuration to the Runner the user has instant access to it without us having to push and update to the helm chart.

Cons

Assuming that we merge !197 (closed) and reject !153 (closed)

  1. We would have to stick to using flags/environment variables for anything we want to override which might not be possible for certain configuration.
  2. Some keys might need to be duplicated since helm chart updates deployment depending on certain configuration for example cache, however this might be fixed if we translate the toml to yaml.
Rejected proposals

Option 1: Go ahead with !153 (closed)

This is what feels most natural to users of a helm chart, they just specify things in the values.yml and then the Runner team will handle the configuration template, go ahead with this we would be able to support volumes so the user just have to specify things in the value.yml. If we go ahead with this we would need to not merge and close !197 (closed) and !228 (closed) since as explained earlier we can't use the configuration template in two places.

Pros

Assuming that we merge !153 (closed) and rejecting !197 (closed) / !228 (closed)

  1. The user is just exposed to the yml configuration and not toml. It seems more natural to use yaml in the kubernetes ecosystem.
  2. The user doesn't have to understand how configuration templates work
  3. This will allow us to add support for any type of configuration inside of the values.yml

Cons

Assuming that we merge !153 (closed) and rejecting !197 (closed) / !228 (closed)

  1. The maintainers of the helm chart are the bottlenecks of adding new features. So users would have to wait for us to add support to something.
  2. We have to keep translating yaml to toml which is not always straight forward.

Rejected reason

I think we all agree here that this is less than ideal because of the cons related to it and it still doens't allow users to specify any configuration for the runner which doesn't improve the current situation that we are in.

Option 3: Translate yaml to toml

We would just translate yaml to toml using the toToml function available so that the user just has to write yaml and it is automatically translated to toml. The generated toml will be used as the configuration template

Pros

  1. Users just use yml which is what they are used too.
  2. Everytime a new configuration is added we don't need to do anything.

Cons

  1. We might translation issues between yaml and toml (need to validate this)

Rejected reason

As tested in #106 (comment 348898590) this won't work because of the float/int translation issue.

Option 4: Hybrid

We can go a hybrid approach where we can use !153 (closed) but then if the user specifies the configuration template we leave it up to the user to configure everything.

Pros

Cons

  1. We end up in a very weird state where something are defined in the values.yml and some aren't. There is no single source of truth the user would have to understand if they need to specify something in the values.yaml or in the configuration template setting. The only way to get around this is to have a breaking change that all configuration should be done inside of the configuration template. This however w
  2. We have to explain how the merging of configuration template works and how it merges things.
  3. Anything that is defined in the values.yml will override the configuration template which might be confusing for the user.
  4. Even more confusing why certain values from the values.yml aren't working because I have the configuration template defined.

Rejected reason

Has no pros, a lot of cons and makes it quite complex and doesn't fulfill the goal that the user can define everything.

Option 5: Support multiple configuration templates

We can add support for the runner where you can specify multiple configuration templates and then we merge them in a specific order. This will be very confusing and hard to debug/develop on.

Rejected reason

Adds complexity that we would have to figure out merging and will cause a lot of issues over which template takes precedence.

Option 6: Have gitlab-runner read yml configuration

Update the [gitlab-runner configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/fc6f8d606f95d90ac35f2b4876b68907c645dc37/common/config.go) read a yamlfile in addition totomlso that we can just use thevalues.yml` struct and save it on disk to for the runner to read.

Pros

  1. Users would be used to writing yaml if a kubernetes environment and they don't have to be exposed to yet another configuration language.
  2. We don't have to translate yaml to toml or the other way round.
  3. Both the helm chart and the gitlab-runner can use a common structure with no translation or duplication.

Cons

  1. Yet another configuration structure for gitlab-runner to support, making sure we check both a config.toml and config.yml file. Which increases the maintainability cost of gitlab-runner
  2. It might be the case a tmol struct doesn't translate well to yml for existing configuration but this needs to be checked.

Issues that will be unblocked

Edited by Darren Eastman
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information