Add graceful-quit to gitlab-ctl

What does this MR do?

Add graceful-quit to gitlab-ctl.

It automates the following process:

1. run sv once service to mark the service as Do not restart if it stops
2. run sv quit service to send SIGQUIT
3. check if the service has stopped within a grace period, by checking if the process group contains any pid
4. do graceful_kill if the service is not stopped

Test plan:

Use the following script to create some dummy services:

• foobar: will print signal and exit
• foobar-stuck: will print signal, but won't exit on SIGQUIT

#!/bin/bash

set -e

BIN_DIR="/opt/gitlab/bin"
LOG_DIR="/var/log/gitlab"
SVC_DIR="/opt/gitlab/sv"
INIT_DIR="/opt/gitlab/init"
RUNIT_DIR="/opt/gitlab/service"

# The foobar service
svc='#!/bin/bash

STUCK_ON_QUIT=${1:-false}

# Function to handle signals
handle_signal() {
    echo "Received signal: $1"
    if [[ $1 == "SIGQUIT" && $STUCK_ON_QUIT == "true" ]]; then
        echo "Stuck on SIGQUIT signal, not exiting"
    else
        exit 0
    fi
}

# Print the PID of the script
echo "Script PID: $$"

if [[ $STUCK_ON_QUIT == "true" ]]; then
    echo "Stuck on SIGQUIT signal"
fi

# Trap various signals and call handle_signal function
trap "handle_signal SIGHUP" HUP
trap "handle_signal SIGINT" INT
trap "handle_signal SIGTERM" TERM
trap "handle_signal SIGQUIT" QUIT
trap "handle_signal SIGUSR1" USR1
trap "handle_signal SIGUSR2" USR2

# Infinite loop to keep the script running
while true; do
    sleep 1
done'

# The runit run script content
svc_run='#!/bin/sh
exec 2>&1

umask 077
exec chpst -P \
  -U root:root \
  -u root:root \
  /opt/gitlab/bin/foobar.sh'

log_run='#!/bin/sh
exec chpst -P \
  -U root:root \
  -u root:root \
  svlogd -tt /var/log/gitlab/' # !! Need to append the service name

function create() {
    echo "$svc" >"$BIN_DIR/foobar.sh"
    chmod +x "$BIN_DIR/foobar.sh"

    svc_name="foobar"
    mkdir -p "$SVC_DIR/$svc_name/log"
    mkdir -p "$LOG_DIR/$svc_name"
    echo "$svc_run" >"$SVC_DIR/$svc_name/run"
    chmod +x "$SVC_DIR/$svc_name/run"
    echo "$log_run$svc_name" >"$SVC_DIR/$svc_name/log/run"
    chmod +x "$SVC_DIR/$svc_name/log/run"
    ln -s "/opt/gitlab/embedded/bin/sv" "$INIT_DIR/$svc_name"
    ln -s "$SVC_DIR/$svc_name" "$RUNIT_DIR/$svc_name"

    svc_name="foobar-stuck"
    mkdir -p "$SVC_DIR/$svc_name/log"
    mkdir -p "$LOG_DIR/$svc_name"
    echo "$svc_run true" >"$SVC_DIR/$svc_name/run"
    chmod +x "$SVC_DIR/$svc_name/run"
    echo "$log_run$svc_name" >"$SVC_DIR/$svc_name/log/run"
    chmod +x "$SVC_DIR/$svc_name/log/run"
    ln -s "/opt/gitlab/embedded/bin/sv" "$INIT_DIR/$svc_name"
    ln -s "$SVC_DIR/$svc_name" "$RUNIT_DIR/$svc_name"

    # Inform the user
    echo "foobar and foobar-stuck services are created"
    echo "foobar will exit on SIGQUIT"
    echo "foobar-stuck will stuck on SIGQUIT"
    echo ""
    echo "Check the logs at"
    echo "/var/log/gitlab/foobar/current"
    echo "/var/log/gitlab/foobar-stuck/current"
}

function cleanup() {
    /opt/gitlab/embedded/bin/sv stop foobar || true
    /opt/gitlab/embedded/bin/sv stop foobar-stuck || true

    rm -rf "$INIT_DIR/foobar"
    rm -rf "$INIT_DIR/foobar-stuck"

    rm -rf "$RUNIT_DIR/foobar"
    rm -rf "$RUNIT_DIR/foobar-stuck"

    rm -rf "$SVC_DIR/foobar"
    rm -rf "$SVC_DIR/foobar-stuck"

    rm -rf "$LOG_DIR/foobar"
    rm -rf "$LOG_DIR/foobar-stuck"

    rm -f "$BIN_DIR/foobar.sh"
    echo "foobar and foobar-stuck services are removed"
}

case "$1" in
create)
    create
    ;;
cleanup)
    cleanup
    ;;
*)
    echo "Usage: $0 {create|cleanup}"
    exit 1
    ;;
esac

Try stopping the dummy services, and check out the logs.

gitlab-ctl graceful-quit foobar
gitlab-ctl graceful-quit foobar-stuck

cat /var/log/gitlab/foobar/current
cat /var/log/gitlab/foobar-stuck/current

Related issues

Closes #8312

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • it is irrelevant
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
  • Hard to set the fixture, as many of the commands do not have unit tests.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

Changelog: added