CNG: Create separate builder images and pipeline

Build tools and dependencies, including Go, can be pulled into separate images and moved out of the the ordinary build pipeline. This is somewhat similar to what we do for Omnibus builder images.

Rationale

These tools and dependencies should not be a part of the final images and their changes are not as frequent as runtime dependencies and build artifacts. For this reason they can have their own lifecycle and iterations.

What is included?

When it comes to tools such as gcc, cmake, or go it is very clear that they need be included in a builder image. They do not change very often and when they do everything needs to be built again.

But for build dependencies the term is too generic and some clarification is in order. Not all build dependencies can be and neither should be included in a builder image. For example, some development libraries that are used for compiling native extensions fit the description but Ruby Gems and Go modules do not.

As a rule of thumb if the build dependencies are version controlled with the final artifact they should not be included in the builder image. This means that we can include most of binary dependencies (for example OpenSSL development headers) and any dependency that doesn’t change with the component that is being built for example libedit where it is missing.

What about the size?

Adding all build tools and dependencies into builder images will make them very large. Since these images do not change frequently their image SHA are more stable when compared to a typical CNG image. Therefore caching can compensate for their large size.

On the other hand, in a multi-stage Dockerfile these images are used for the build stage and since they come with the majority of tools and dependencies, the build stage focuses on building the artifact rather than installing requirements which will speed up the process.

Considerations

That being said, we need a guideline for tagging and versioning of these images so that the CNG images can reference them without ambiguity, possibly via a build argument.

We also need different builder images for each image flavor, i.e. Debian and UBI, per architecture.