workhorse: Support generating configuration via external command

What does this MR do and why?

This commit adds a config_command option that allows an admin to pull in configuration values for the config.toml.

In the context of FIPS we need a mechanism that allows us to not store any credentials in plaintext format on-disk. Ideally, we would hook into a solution like AWS Secrets or Kubernetes Secrets that lets us retrieve the secrets at startup time so that they don't have to exist on disk in any form at all. But there exists a bunch of different solutions for secrets management, where many make sense in one context but don't in any other contexts.

So implementing support for e.g. AWS Secrets directly would put us into a position where we now need to argue which solutions we want to support and which we don't. This shows that directly supporting any of them directly is not the right way to go, but that we should instead support a general schema that allows administrators to easily plug in whatever they are using without too much of a hassle.

A simple and boring solution to this is to implement support for running an external command on startup that generates the secrets for us. Like this, an administrator can simply provide a script that runs e.g. aws get-secret-value and Workhorse would know to put the secrets into the correct spot in our configuration. But that again has another issue: there are several different fields that may contain secrets, and making sure they all can be adjusted might be tedious.

To fix that, we simply generalize the idea: why only care about secrets, when we can instead provide a mechanism that allows the external command to generate the complete configuration which we then merge back into the initial configuration read from disk?

The implementation of this is straight-forward: we read the initial configuration from disk, and when a specific key is set we use its value as the executable that is to be spawned. We expect the executable to generate JSON, which we then deserialize and thus merge into the config structure we have already parsed. Like this, the command is free to only generate configuration at runtime that it wants to actually change, and all the other configuration would stay as-is.

This follows the pattern in gitaly!5525 (merged) and gitaly!6157 (merged).

Relates to:

• omnibus-gitlab#6275 (closed)
• #435183 (closed)

How to set up and validate locally

I used this simple test to change the Redis DB from 0 to 10.

1. In the gitlab/workhorse/config.toml, I added:

config_command = "/tmp/workhorse-config.rb"

2. In /tmp/workhorse-config.rb I created an executable script:

#!/usr/bin/env ruby
require 'json'
puts JSON.dump(redis: { db: 10 })

3. To verify Workhorse is using DB 10, from my GDK directory I ran:

redis-cli -s redis/redis.socket monitor | grep "\[10 "

4. Then I ran gdk restart gitlab-workhorse and saw:

1704307116.526705 [10 unix:/Users/stanhu/gdk-ee/redis/redis.socket] "select" "10"
1704307116.526750 [10 unix:/Users/stanhu/gdk-ee/redis/redis.socket] "ping"