Skip to content

gitaly: Fix cgroups v2 configuration issues

James Liu requested to merge jliu/gitaly-cgroupsv2 into master

What does this MR do?

This MR addresses configuration issues in Omnibus that prevent users from enabling cgroups for Gitaly when running on a cgroups v2-enabled system. The problems are documented in the following issues:

The commit messages elaborate on the required changes too.

Test plan

This test plan verifies that Gitaly is able to assign child git processes to a cgroup. The steps should be repeated with a cgroup v1 and a cgroup v2-enabled system. We'll use Ubuntu 20.04 (cgroup v1) and Ubuntu 22.04 (cgroup v2).

  1. Download the .deb package generated by the package jobs:
    • Ubuntu 22.04, cgroups v2: curl -o gitlab.deb --location --header "PRIVATE-TOKEN: <REDACTED>" "https://gitlab.com/api/v4/projects/gitlab-org%2Fomnibus-gitlab/jobs/8074212311/artifacts/pkg/ubuntu-jammy/gitlab-ee_17.4.2+rfbranch.1494308969.5e97983e-0_amd64.deb"
    • Ubuntu 20.04, cgroups v1: curl -o gitlab.deb --location --header "PRIVATE-TOKEN: <REDACTED>" "https://gitlab.com/api/v4/projects/gitlab-org%2Fomnibus-gitlab/jobs/8074258497/artifacts/pkg/ubuntu-focal/gitlab-ee_17.4.2+rfbranch.1494318855.5e97983e-0_amd64.deb"
  2. If installing on top of an existing installation, run sudo systemctl daemon-reload to ensure the runsv unit file is reloaded, and that systemd configures cgroup delegation appropriately.
  3. Install the package: sudo EXTERNAL_URL="http://localhost" apt install ./gitlab.deb
  4. Log in and verify that a project/repository can be created.
  5. Modify /etc/gitlab/gitlab.rb to enable cgroups for Gitaly. We do this by setting cgroups.repositories.count to a positive number. We'll also configure some arbitrary resource limits, and confirm they've been set correctly in the cgroups hierarchy:
gitaly['configuration'] = {
      cgroups: {
          memory_bytes: 6442450944,
          cpu_shares: 1024,
          cpu_quota_us: 200000,
          repositories: {
              count: 10,
              memory_bytes: 4294967296,
              cpu_shares: 512,
              cpu_quota_us: 100000
          }
      }
}
  1. sudo gitlab-ctl reconfigure.
  2. Review /var/opt/gitlab/gitaly/config.toml and ensure the following block is present:
[cgroups]
memory_bytes = 6442450944
cpu_shares = 1024
cpu_quota_us = 200000
mountpoint = "/sys/fs/cgroup"
hierarchy_root = "gitlab.slice/gitaly"

  [cgroups.repositories]
  count = 10
  memory_bytes = 4294967296
  cpu_shares = 512
  cpu_quota_us = 100000
  1. Repeat step 3.
  2. We'll now validate that resource limits have been configured correctly in the cgroup hierarchy. sudo bash to obtain a root shell, and then verify the following files according to the system you're currently testing:

cgroups v1

File Expected value
/sys/fs/cgroup/cpu/gitlab.slice/gitaly/gitaly-PID/cpu.shares 1024
/sys/fs/cgroup/cpu/gitlab.slice/gitaly/gitaly-PID/cpu.cfs_quota_us 200000
/sys/fs/cgroup/cpu/gitlab.slice/gitaly/gitaly-PID/repos-0/cpu.shares 512
/sys/fs/cgroup/cpu/gitlab.slice/gitaly/gitaly-PID/repos-0/cpu.cfs_quota_us 100000
/sys/fs/cgroup/memory/gitlab.slice/gitaly/gitaly-PID/memory.limit_in_bytes 6442450944
/sys/fs/cgroup/memory/gitlab.slice/gitaly/gitaly-PID/repos-0/memory.limit_in_bytes 4294967296

cgroups v2

File Expected value
/sys/fs/cgroup/gitlab.slice/gitaly/gitaly-PID/cpu.max 200000 100000
/sys/fs/cgroup/gitlab.slice/gitaly/gitaly-PID/cpu.weight 39 (conversion is performed by a library, and weights are relative to other defined weights)
/sys/fs/cgroup/gitlab.slice/gitaly/gitaly-PID/memory.max 6442450944
/sys/fs/cgroup/gitlab.slice/gitaly/gitaly-PID/repos-0/cpu.max 200000 100000
/sys/fs/cgroup/gitlab.slice/gitaly/gitaly-PID/repos-0/cpu.weight 20 (approx. half of the previous weight)
/sys/fs/cgroup/gitlab.slice/gitaly/gitaly-PID/repos-0/memory.max 4294967296

Related issues

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by James Liu

Merge request reports