Skip to content

Update Puma to 6.4.3 and patch PID reaper

Stan Hu requested to merge sh-update-and-patch-puma into master

What does this MR do and why?

Puma v6.4.1 was reverted in #437948 (closed) since it exposed a PID-reaping bug in Ruby 3.1.4 (https://bugs.ruby-lang.org/issues/19837). Full details are in https://github.com/puma/puma/issues/3313.

Unfortunately Ruby 3.1.5 and 3.2.5 still have a bug that prevents Puma from reaping child PIDs properly (https://bugs.ruby-lang.org/issues/20490), so let's pull in the patch in https://github.com/puma/puma/pull/3314 since a Puma release has not been forthcoming.

This update fixes a minor security issue and other bugs: https://my.diffend.io/gems/puma/6.4.0/6.4.3

Related incidents:

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

GDK

  1. Check out this branch.
  2. gdk restart rails-web
  3. When Puma starts, kill one of the cluster worker jobs:
% ps -ef | grep "cluster worker"
  501 65022 64509   0 10:08PM ??         0:00.38 puma: cluster worker 0: 64509 [gitlab-puma-worker]
  501 65378 64509   0 10:08PM ??         0:00.40 puma: cluster worker 1: 64509 [gitlab-puma-worker]
  501 79824  7003   0 10:17PM ttys004    0:00.00 grep cluster worker
% kill -9 65378

You should see a new PID take its place immediately:

 % ps -ef | grep "cluster worker"
  501 65022 64509   0 10:08PM ??         0:00.39 puma: cluster worker 0: 64509 [gitlab-puma-worker]
  501 80175 64509   0 10:18PM ??         0:00.37 puma: cluster worker 1: 64509 [gitlab-puma-worker]
  501 80213  7003   0 10:18PM ttys004    0:00.00 grep cluster worker

With Cloud Native GitLab

  1. Build a webservice container with Puma 6.4.3. I used this Dockerfile:
FROM registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:master

USER root
RUN apt update && apt install -y build-essential && cd /srv/gitlab && sed -i -e "s/= 6.4.0/= 6.4.3/" Gemfile && bundle config set frozen false && bundle install && bundle config set frozen true

COPY config/initializers/puma_patch.rb /srv/gitlab/config/initializers/puma_patch.rb

USER git
  1. Push this image to some public location. For example:
docker build -t registry.gitlab.com/stanhu/lfs-test/webservice-ee:latest .
docker push registry.gitlab.com/stanhu/lfs-test/webservice-ee:latest

If you are running this from your gitlab dir, be aware that .dockerignore may interfere with the above.

  1. Install the Helm Chart with this image:
helm upgrade --install gitlab . \
     --set gitlab.webservice.image.pullPolicy=Always \
     --set gitlab.webservice.image.repository=registry.gitlab.com/stanhu/lfs-test/webservice-ee \
     --set gitlab.webservice.image.tag=latest
  1. When the webservice pod comes up, log into it and verify that Puma 6.4.3 is running:
git@gitlab-webservice-default-66f688f648-g6prb:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
git            1       0 87 20:09 ?        00:00:39 puma 6.4.2 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           21       1  0 20:09 ?        00:00:00 /usr/local/bin/gitlab-logger /var/log/gitlab
git           68       0  0 20:09 pts/0    00:00:00 bash
git           81       1 65 20:09 ?        00:00:02 ruby /srv/gitlab/bin/metrics-server
git           84       1 22 20:09 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           86       1 16 20:09 ?        00:00:00 puma: cluster worker 1: 1 [gitlab-puma-worker]
git          121      68  0 20:09 pts/0    00:00:00 ps -ef
  1. Then start killing the cluster worker PIDs and verify they are reaped. Without the patch, you will see a zombie process:
git@gitlab-webservice-default-66f688f648-g6prb:/$ kill -9 86
git@gitlab-webservice-default-66f688f648-g6prb:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
git            1       0 45 20:09 ?        00:00:39 puma 6.4.2 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           21       1  0 20:09 ?        00:00:00 /usr/local/bin/gitlab-logger /var/log/gitlab
git           68       0  0 20:09 pts/0    00:00:00 bash
git           81       1  4 20:09 ?        00:00:02 ruby /srv/gitlab/bin/metrics-server
git           84       1  1 20:09 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           86       1  1 20:09 ?        00:00:00 [ruby] <defunct>
git          127      68  0 20:10 pts/0    00:00:00 ps -ef

With the patch, it works:

git@gitlab-webservice-default-6887959876-mg9mb:/srv/gitlab/config/initializers$ ps -ef | grep puma
git            1       0 88 05:12 ?        00:00:32 puma 6.4.3 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           53       1 19 05:13 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           55       1 19 05:13 ?        00:00:00 puma: cluster worker 1: 1 [gitlab-puma-worker]
git           89      35  0 05:13 pts/0    00:00:00 grep puma
git@gitlab-webservice-default-6887959876-mg9mb:/srv/gitlab/config/initializers$ kill 55
git@gitlab-webservice-default-6887959876-mg9mb:/srv/gitlab/config/initializers$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
git            1       0 80 05:12 ?        00:00:32 puma 6.4.3 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           21       1  0 05:12 ?        00:00:00 /usr/local/bin/gitlab-logger /var/log/gitlab
git           35       0  0 05:13 pts/0    00:00:00 bash
git           50       1 29 05:13 ?        00:00:01 ruby /srv/gitlab/bin/metrics-server
git           53       1  7 05:13 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           92       1 29 05:13 ?        00:00:00 puma: cluster worker 1: 1 [gitlab-puma-worker]
git          109      35  0 05:13 pts/0    00:00:00 ps -ef
Edited by Stan Hu

Merge request reports