Introduce Ubuntu-flavored runner-helper image

Using Alpine as a base distribution for docker images as several advantages: the image is smaller and generally has a reduced security footprint. It also has disadvantages: using musl libc instead of glibc, which is more common. musl libc is famous for causing hard to troubleshoot DNS issues. For example, one of our customers reported:

Our K8s team strongly recommends avoiding Alpine based images due to problems with DNS:

Avoid Alpine / MUSL libc - Container images running on K8s SHOULD NOT be based on Alpine or use the musl libc. The reason is that musl libc's DNS resolver results dropped packets when running on K8s. To an application, this looks like slow DNS resolution. This is true on every K8s distribution whether Nordstrom's shared clusters, GKE, EKS, AKS, etc.

Which correlates well with the symptoms we are having. Would it be possible to provide a Debian-based helper and runner images as part of the set of images published with gitlab-runner? We looked for those in the registry and did not see any.

An example of a customer issue is here #4129 (closed) with further discussion

The following blog posts have further information:

• https://www.adammargherio.com/a-perfect-dns-storm/
• https://andydote.co.uk/2019/12/30/consul-alpine-dns-revisited/

After discussing with the team (slack thread), we feel that Alpine's smaller size is valuable, and we wouldn't want to switch to our Ubuntu image as a default. However, Google's distroless has many of the same advantages but is based on glibc, with an even smaller footprint.

Using Distroless instead of Alpine as a default would then solve these weird DNS problems while keeping a small size and footprint.

For gitlab-runner-helper, we need to install a full shell environment to run commands when cloning or managing the cache. For this image, we then still need a base distribution able to download and install packages. Switching to ubuntu then has the advantage of solving the DNS issue, at the cost of a slight increase in size. I believe this is the right tradeoff given the flaky and unpredictable issues with musl, making it hard for the user to identify and troubleshoot.

Proposal

May 6th, 2021 Update:

Rather than using distroless, we've decided to use ubuntu and not worry so much about the image sizes. The compressed image size doesn't appear to be much of a problem.

gitlab-runner-helper will now have two flavors: ubuntu and alpine, with alpine being the default until some time in the future.

Eventually, we'll introduce a multiarch runner-helper image, and this will default to using ubuntu for increased compatibility.