Use builder images (!1306) · Merge requests · GitLab.org / Build / CNG

Hossein Pursultani requested to merge hp-use-builder-image into master Feb 28, 2023

What does this MR do?

⚠ Do not merge before !1308 (merged).

This MR refactors the build stage of all the container images to use the builder images that are introduced in !1308 (merged).

Overview

Major changes

gitlab-ubi-builder and gitlab-go images are removed. They are replaced by builder images from !1308 (merged).
Builder images are used in the build stage of all container images (with a few exceptions, including gitlab-base).
Installation instructions for build tools and development headers are removed from build stages.
Some instructions, including downloading and verifying source and binary artifacts, are replaced with fetch and gitlab-fetch helper scripts.
The download-dependencies-build scripts are removed.
As a result of these changes, the build stage of most Debian and UBI images look the same. This may seem repetitive but it is done intentionally. In a follow-up MR we will remove the flavored Dockefiles and combine them into a single default Dockerfile to reduce maintenance effort.
The build stage of gitlab-rails/Dockerfile.erb is refactored significantly, leveraging the new possibilities that are available with builder images:
- It is no longer a template and is a plain Dockerfile. The parametric behavior for EE and asset compilation is replaced with two workarounds: 1) Bash if statements for conditional execution when EE is enabled or COMPILE_ASSETS is true; 2) Using scratch image and passing it as a build argument to GITLAB_ELASTICSEARCH_INDEXER_IMAGE and GITLAB_ASSETS_IMAGE (the COPY command does not copy anything and creates an empty layer).
- More steps from the final image are pulled into the build stage to eliminate the need for chown and chmod. For example initializing bootsnap.
The build stages are de-optimized by breaking down large RUN commands into multiple RUN commands and grouping together relevant instructions. Also, ARGs are brought closer to where they are used. These are to increase the chance to use cache an reduce the possibility of rebuilding unnecessary steps.

Minor changes

A more consistent naming scheme is adopted for build arguments. For example instead of passing components of an image reference to a Dockerfile the whole image is passed as a build argument. Another example are the operational parameters, e.g. API_KEY, CURL_OPTS, etc.
Signatures for all resources (both binary and source) that are downloaded from external sources are verified. For resources that the signature can not be obtained automatically the SHA256 signature is declared as CI variables and used as Docker build arguments.
The layout of the final asset images are changed so that the keep the files in the intended directory tree. This means that when other images are copying the assets they don't need to place them in the right place. A simple COPY --from=asset / / will do the job.

Verification

We can use container-diff for static verification. With its output, we can identify the files that are changed before and after refactoring. Here is a example script:

#!/usr/bin/env bash

set -euxo pipefail

SOURCE_TAG='master'
TARGET_TAG='hp-use-builder-image'

REGISTRY='registry.gitlab.com/gitlab-org/build/cng'

IMAGES=(
    # asset images
    gitlab-rust
    gitlab-gomplate
    gitlab-graphicsmagick
    gitlab-logger
    gitlab-python
    postgresql
    gitlab-exiftool
    gitlab-elasticsearch-indexer
    gitlab-metrics-exporter

    # intermediate images
    gitlab-base
    gitlab-ruby
    gitlab-rails-ee
    gitlab-rails-ce

    # runtime images
    kubectl
    gitaly
    gitlab-container-registry
    gitlab-exporter
    gitlab-kas
    gitlab-mailroom
    gitlab-pages
    gitlab-shell
    gitlab-zoekt-dynamic-indexer
    gitlab-zoekt-webserver

    # rails-based runtime images
    gitlab-geo-logcursor
    gitlab-sidekiq-ce
    gitlab-sidekiq-ee
    gitlab-toolbox-ce
    gitlab-toolbox-ee
    gitlab-webservice-ce
    gitlab-webservice-ee
    gitlab-workhorse-ce
    gitlab-workhorse-ee
)

for _image in "${IMAGES[@]}"; do
    container-diff diff \
        "${REGISTRY}/${_image}:${TARGET_TAG}" \
        "${REGISTRY}/${_image}:${SOURCE_TAG}" \
        --type=apt --type=file | tee "${_image}.txt"
done

WIP: Summary of `container-diff` (with master) for Debian-based images

✅ `gitlab-rust`

✅ It is based on scratch. All base OS files are removed.
✅ Rust assets, i.e. /usr/local/{bin,etc,lib,libexec,share} directories, are moved from /assets to /. The source path of COPY command in gitlab-ruby and gitlab-rails images are adjusted to match the change. This is done to use the COPY --from=... / / command style for asset images.

NOTE: This image can be moved to builder images.

✅ `gitlab-gomplate`

✅ No change.

✅ `gitlab-graphicsmagick`

✅ No change.

✅ `gitlab-logger`

✅ No change.

⚠ `gitlab-python`

✅ It is based on scratch. All base OS files are removed.
✅ All Python assets are kept in the same directory structure, i.e. /usr/local/{bin,include,lib} directories.
✅ All Python assets are moved from /assets to / instead. The source path of COPY command in gitlab-rails images are adjusted to match the change. This is done to use the COPY --from=... / / command style for asset images.
❗ Additional modules with binary extensions are added, for example bz2, curses, lzma, and uuid, with the total size of less than 1MB. This is the result of the availability of the development headers of these libraries in the builder image. configure script has detected the headers and built the libraries. To avoid that we can investigate the possibility of disabling them with configure script.

✅ `postgresql`

✅ PostgreSQL assets are moved from /usr/local/psql to /usr. This is the place that gitlab-rails image expects them. The source path of the COPY command in gitlab-rails image is adjusted. This is done to reduce the number of COPY commands and rewrite simplify the layering of asset images. As a result we can see that most COPY commands from asset images look like this COPY --from=postgresql / /.

✅ `gitlab-exiftool`

✅ No change.

✅ `gitlab-elasticsearch-indexer`

✅ It is based on scratch. All base OS files are removed.
✅ gitlab-elasticsearch-indexer executable is moved to /usr/bin/gitlab-elasticsearch-indexer. This is the place that gitlab-rails image expects it. The COPY command in gitlab-rails image is adjusted and changed to COPY --from=gitlab-elasticsearch-indexer / /.

✅ `gitlab-metrics-exporter`

✅ gitlab-metrics-exporter executable binary is moved to /usr/bin/gitlab-metrics-exporter. This is the place that gitlab-rails image expects it. The COPY command in gitlab-rails image is adjusted and changed to COPY --from=gitlab-metrics-exporter / /.

✅ `gitlab-base`

✅ No change.

⚠ `gitlab-ruby`

✅ The following packages are removed in the Debian-based image:

NAME                   VERSION                        SIZE
-bzip2                 1.0.8-4                        122K
-libc-dev-bin          2.31-13 deb11u5                377K
-libc6-dev             2.31-13 deb11u5                14.2M
-libcrypt-dev          1:4.4.18-4                     317K
-libffi-dev            3.3-6                          301K
-libnsl-dev            1.3.0-2                        345K
-libssl-dev            1.1.1n-0 deb11u4               7.8M
-libtirpc-dev          1.3.1-1 deb11u1                723K
-libyaml-dev           0.2.2-1                        250K
-linux-libc-dev        5.10.162-1                     5.7M
-zlib1g-dev            1:1.2.11.dfsg-2 deb11u2        578K

Their development headers, object files, shared objects, and executables are removed.

✅ The following packages are added to the Debian-based image:

+libbsd0         0.11.3-1                 191K
+libedit2        3.1-20191231-2 b1        259K
+libmd0          1.0.3-3                  77K

✅ /root/.local gem cache and /build-scripts directories are removed.
✅ Some libraries that were installed as the transient dependencies of the -dev packages had been missing, including libedit2. They are now installed explicitly.
❗ Some Ruby Gem executables, such as racc, rbs, and ri, are removed from /usr/bin directory.

⚠ `kubectl`

✅ The following packages are removed from Debian-base image:

NAME                        VERSION                               SIZE
-curl                       7.74.0-1.3 deb11u7                    430K
-libbrotli1                 1.0.9-2 b2                            784K
-libcurl4                   7.74.0-1.3 deb11u7                    740K
-libldap-2.4-2              2.4.57 dfsg-3+deb11u1                 539K
-libnghttp2-14              1.43.0-1                              217K
-libpsl5                    0.21.0-1.2                            95K
-librtmp1                   2.4 20151223.gitfa8646d.1-2+b2        146K
-libsasl2-2                 2.1.27 dfsg-2.1+deb11u1               188K
-libsasl2-modules-db        2.1.27 dfsg-2.1+deb11u1               101K
-libssh2-1                  1.9.0-2                               298K

This is because with the builder we don't need curl command anymore. Theire shared objects and executables are removed.

❗ /licenses/GitLab.txt is added to the Debian-based image. This is because UBI and Debian build instructions are unified.

✅ `gitlab-container-registry`

✅ No change.

✅ `gitlab-pages`

✅ No change.

✅ `gitlab-shell`

✅ No change.

⚠ `gitlab-kas`

✅ No meaningful change.
❗ /licenses/GitLab.txt is added to the Debian-based image. This is because UBI and Debian build instructions are unified.

✅ `gitlab-zoekt-dynamic-indexserver`

✅ No change.

✅ `gitlab-zoekt-webserver`

✅ No change.

⚠ `gitaly`

⚠ Changes from gitlab-ruby.

✅ The following packages are removed from the Debian-based image (some of them are removed in gitlab-ruby):

NAME                   VERSION                        SIZE
-libc-dev-bin          2.31-13 deb11u5                377K
-libc6-dev             2.31-13 deb11u5                14.2M
-libcrypt-dev          1:4.4.18-4                     317K
-libffi-dev            3.3-6                          301K
-libnsl-dev            1.3.0-2                        345K
-libssl-dev            1.1.1n-0 deb11u4               7.8M
-libtirpc-dev          1.3.1-1 deb11u1                723K
-libyaml-dev           0.2.2-1                        250K
-linux-libc-dev        5.10.162-1                     5.7M
-zlib1g-dev            1:1.2.11.dfsg-2 deb11u2        578K

Their development headers, object files, shared objects, and executables are removed.

✅ bench directory is removed from msgpack-1.3.3 Gem. The directory contains benchmark tests. It is cleaned up by cleanup-gems.
❗ /licenses/GitLab.txt is added to the Debian-based image. This is because UBI and Debian build instructions are unified.

⚠ `gitlab-exporter`

⚠ Changes from gitlab-ruby.

✅ The following packages are removed in the Debian-based image:

NAME                   VERSION                        SIZE
-bzip2                 1.0.8-4                        122K
-libc-dev-bin          2.31-13 deb11u5                377K
-libc6-dev             2.31-13 deb11u5                14.2M
-libcrypt-dev          1:4.4.18-4                     317K
-libffi-dev            3.3-6                          301K
-libnsl-dev            1.3.0-2                        345K
-libpq-dev             13.9-0 deb11u1                 555K
-libpq5                13.9-0 deb11u1                 790K
-libssl-dev            1.1.1n-0 deb11u4               7.8M
-libtirpc-dev          1.3.1-1 deb11u1                723K
-libyaml-dev           0.2.2-1                        250K
-linux-libc-dev        5.10.162-1                     5.7M
-zlib1g-dev            1:1.2.11.dfsg-2 deb11u2        578K

Their development headers, object files, shared objects, and executables are removed.

✅ The following packages are added to the Debian-based image:

+libbsd0         0.11.3-1                 191K
+libedit2        3.1-20191231-2 b1        259K
+libmd0          1.0.3-3                  77K

✅ Other changes from gitlab-ruby.
✅ bench directory is removed from mustermann-2.0.2 Gem. The directory contains benchmark tests. It is cleaned up by cleanup-gems.
❗ Some Ruby Gem executables, such as puma, pumactl, and rackup are removed from /usr/bin directory.

⚠ `gitlab-mailroom`

⚠ Changes from gitlab-ruby.

✅ The following packages are removed in the Debian-based image:

NAME                   VERSION                        SIZE
-binutils                         2.35.2-2                       97K
-binutils-common                  2.35.2-2                       13.6M
-binutils-x86-64-linux-gnu        2.35.2-2                       8.5M
-bzip2                            1.0.8-4                        122K
-cpp                              4:10.2.1-1                     42K
-cpp-10                           10.2.1-6                       25M
-g++                              4:10.2.1-1                     15K
-g++-10                           10.2.1-6                       26.8M
-gcc                              4:10.2.1-1                     45K
-gcc-10                           10.2.1-6                       49.9M
-icu-devtools                     67.1-7                         680K
-libasan6                         10.2.1-6                       9.4M
-libatomic1                       10.2.1-6                       45K
-libbinutils                      2.35.2-2                       2.8M
-libc-dev-bin                     2.31-13 deb11u5                377K
-libc6-dev                        2.31-13 deb11u5                14.2M
-libcc1-0                         10.2.1-6                       148K
-libcrypt-dev                     1:4.4.18-4                     317K
-libctf-nobfd0                    2.35.2-2                       191K
-libctf0                          2.35.2-2                       141K
-libdpkg-perl                     1.20.12                        2.5M
-libffi-dev                       3.3-6                          301K
-libgcc-10-dev                    10.2.1-6                       13.2M
-libgdbm-compat4                  1.19-2                         67K
-libgomp1                         10.2.1-6                       278K
-libicu67                         67.1-7                         32.4M
-libisl23                         0.23-1                         2.2M
-libitm1                          10.2.1-6                       111K
-liblsan0                         10.2.1-6                       3.1M
-libmpc3                          1.2.0-1                        129K
-libmpfr6                         4.1.0-3                        3.3M
-libnsl-dev                       1.3.0-2                        345K
-libperl5.32                      5.32.1-4 deb11u2               27.2M
-libquadmath0                     10.2.1-6                       300K
-libssl-dev                       1.1.1n-0 deb11u4               7.8M
-libstdc++-10-dev                 10.2.1-6                       17.3M
-libtirpc-dev                     1.3.1-1 deb11u1                723K
-libtsan0                         10.2.1-6                       8.7M
-libubsan1                        10.2.1-6                       2.9M
-libyaml-dev                      0.2.2-1                        250K
-linux-libc-dev                   5.10.162-1                     5.7M
-patch                            2.7.6-7                        248K
-perl                             5.32.1-4 deb11u2               705K
-perl-modules-5.32                5.32.1-4 deb11u2               17M
-xz-utils                         5.2.5-2.1~deb11u1              612K
-zlib1g-dev                       1:1.2.11.dfsg-2 deb11u2        578K

Their development headers, object files, shared objects, and executables are removed.

✅ The following packages are added to the Debian-based image:

+libbsd0         0.11.3-1                 191K
+libedit2        3.1-20191231-2 b1        259K
+libmd0          1.0.3-3                  77K

✅ The final image size is reduced to 272MB from 630MB.
✅ Other changes from gitlab-ruby.
✅ Unused /scripts/healthcheck is removed.
✅ webrick-1.7.0 Gem which is a development dependency is removed.

⚠ `gitlab-rails-ce`

⚠ Changes from gitlab-ruby.

✅ The following packages are removed from the Debian-based image (some of them are removed in gitlab-ruby):

-bzip2                 1.0.8-4                        122K
-icu-devtools          67.1-7                         680K
-libbsd-dev            0.11.3-1                       735K
-libc-dev-bin          2.31-13 deb11u5                377K
-libc6-dev             2.31-13 deb11u5                14.2M
-libcrypt-dev          1:4.4.18-4                     317K
-libedit-dev           3.1-20191231-2 b1              496K
-libffi-dev            3.3-6                          301K
-libicu-dev            67.1-7                         43.2M
-libmd-dev             1.0.3-3                        205K
-libncurses-dev        6.2 20201114-2                 2.3M
-libnsl-dev            1.3.0-2                        345K
-libre2-9              20210201 dfsg-1                487K
-libre2-dev            20210201 dfsg-1                1.1M
-libssl-dev            1.1.1n-0 deb11u4               7.8M
-libtirpc-dev          1.3.1-1 deb11u1                723K
-libxml2-dev           2.9.10 dfsg-6.7+deb11u3        3.2M
-libyaml-dev           0.2.2-1                        250K
-linux-libc-dev        5.10.162-1                     5.7M
-sudo                  1.9.5p2-3 deb11u1              4.5M
-zlib1g-dev            1:1.2.11.dfsg-2 deb11u2        578K

Their development headers, object files, shared objects, and executables are removed.

✅ Since the the image is built with the latest source from master branch, there could be changes in Rails app and assets.
✅ /srv/gitlab/haml_lint directory is removed`.
✅ Since bundler is updated in gitlab-ruby, there is no excess for bundler in /srv/gitlab/vendor/bundle.
🔧 [FIXED] ~~❌ The new image contains EE assets. Source of the problem is a typo: !1306 (comment 1341957777)~~
🔧 [FIXED] ~~❌ Locale assets are missing.~~
🔧 [FIXED] ~~❌ The new image contains 2.7.0 gems: !1306 (comment 1343104042)~~
🔧 [FIXED] ~~cleanup-gems does not remove all mkmf.log and gem_make.out files.~~
🔧 [FIXED] ~~All .* files are removed from /srv/gitlab.~~

⚠ `gitlab-rails-ee`

⚠ Changes from gitlab-ruby.

✅ The following packages are removed from the Debian-based image (some of them are removed in gitlab-ruby):

-bzip2                 1.0.8-4                        122K
-icu-devtools          67.1-7                         680K
-libbsd-dev            0.11.3-1                       735K
-libc-dev-bin          2.31-13 deb11u5                377K
-libc6-dev             2.31-13 deb11u5                14.2M
-libcrypt-dev          1:4.4.18-4                     317K
-libedit-dev           3.1-20191231-2 b1              496K
-libffi-dev            3.3-6                          301K
-libicu-dev            67.1-7                         43.2M
-libmd-dev             1.0.3-3                        205K
-libncurses-dev        6.2 20201114-2                 2.3M
-libnsl-dev            1.3.0-2                        345K
-libre2-9              20210201 dfsg-1                487K
-libre2-dev            20210201 dfsg-1                1.1M
-libssl-dev            1.1.1n-0 deb11u4               7.8M
-libtirpc-dev          1.3.1-1 deb11u1                723K
-libxml2-dev           2.9.10 dfsg-6.7+deb11u3        3.2M
-libyaml-dev           0.2.2-1                        250K
-linux-libc-dev        5.10.162-1                     5.7M
-sudo                  1.9.5p2-3 deb11u1              4.5M
-zlib1g-dev            1:1.2.11.dfsg-2 deb11u2        578K

Their development headers, object files, shared objects, and executables are removed.

✅ Since the the image is built with the latest source from master branch, there could be changes in Rails app and assets.
✅ /srv/gitlab/haml_lint directory is removed`.
✅ Since bundler is updated in gitlab-ruby, there is no excess for bundler in /srv/gitlab/vendor/bundle.
🔧 [FIXED] ~~❌ Locale assets are missing.~~
🔧 [FIXED] ~~❌ The new image contains 2.7.0 gems: !1306 (comment 1343104042)~~
🔧 [FIXED] ~~cleanup-gems does not remove all mkmf.log and gem_make.out files.~~
🔧 [FIXED] ~~All .* files are removed from /srv/gitlab.~~

Related issues

Closes:

Related to:

gitlab-org/charts/gitlab#3407 (moved)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

Merge Request Title, and Description are up to date, accurate, and descriptive
MR targeting the appropriate branch
MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

Test plan indicating conditions for success has been posted and passes
Documentation created/updated
Integration tests added to GitLab QA
The impact any change in container size has should be evaluated

Edited Apr 13, 2023 by Hossein Pursultani

Use builder images

What does this MR do?

Overview

Major changes

Minor changes

Verification

WIP: Summary of container-diff (with master) for Debian-based images

✅ gitlab-rust

✅ gitlab-gomplate

✅ gitlab-graphicsmagick

✅ gitlab-logger

⚠ gitlab-python

✅ postgresql

✅ gitlab-exiftool

✅ gitlab-elasticsearch-indexer

✅ gitlab-metrics-exporter

✅ gitlab-base

⚠ gitlab-ruby

⚠ kubectl

✅ gitlab-container-registry

✅ gitlab-pages

✅ gitlab-shell

⚠ gitlab-kas

✅ gitlab-zoekt-dynamic-indexserver

✅ gitlab-zoekt-webserver

⚠ gitaly

⚠ gitlab-exporter

⚠ gitlab-mailroom

⚠ gitlab-rails-ce

⚠ gitlab-rails-ee

Related issues

Checklist

Required

Expected (please provide an explanation if not completing)

Merge request reports

WIP: Summary of `container-diff` (with master) for Debian-based images

✅ `gitlab-rust`

✅ `gitlab-gomplate`

✅ `gitlab-graphicsmagick`

✅ `gitlab-logger`

⚠ `gitlab-python`

✅ `postgresql`

✅ `gitlab-exiftool`

✅ `gitlab-elasticsearch-indexer`

✅ `gitlab-metrics-exporter`

✅ `gitlab-base`

⚠ `gitlab-ruby`

⚠ `kubectl`

✅ `gitlab-container-registry`

✅ `gitlab-pages`

✅ `gitlab-shell`

⚠ `gitlab-kas`

✅ `gitlab-zoekt-dynamic-indexserver`

✅ `gitlab-zoekt-webserver`

⚠ `gitaly`

⚠ `gitlab-exporter`

⚠ `gitlab-mailroom`

⚠ `gitlab-rails-ce`

⚠ `gitlab-rails-ee`