Following docs for *base_dir config options causes K8s to reject job pods
Summary
Currently, setting logs_base_dir
and scripts_base_dir
both to /tmp
for example (as the documentation suggests), results in the runner failing to create job pods.
Steps to reproduce
Deploy GitLab runners using the k8s executor. Set runner config options as outlined here
.gitlab-ci.yml
default:
image: quay.io/helmpack/chart-testing:latest
tags:
- homelab
chart:lint:
variables:
ALL_CHARTS: ${ALL_CHARTS:-"false"}
CLUSTER_NAME: ${CI_PIPELINE_ID}-${K8S_VERSION}
GIT_DEPTH: 50
GIT_STRATEGY: clone
before_script:
- git remote -v
- git fetch
script:
- |
CHANGED_CHARTS=$(ct list-changed --config .gitlab/ct.yaml)
CT_FLAGS=(--config .gitlab/ct.yaml --lint-conf .gitlab/lint.yaml --helm-repo-extra-args "wojoinc-charts=--username gitlab-ci-token --password ${CI_JOB_TOKEN}")
if [[ $ALL_CHARTS == "true" ]]; then
CT_FLAGS+=(--all)
fi
echo "Detected changes in charts:"
echo "${CHANGED_CHARTS[*]}"
ct lint "${CT_FLAGS[@]}"
echo "CHANGED_CHARTS=$CHANGED_CHARTS" >> variables.env
echo "Printing env file"
cat variables.env
artifacts:
reports:
dotenv: variables.env
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
changes:
paths:
- charts/*/Chart.yaml
Actual behavior
Instead of producing paths like /tmp/logs-52717018-7386788331
and /tmp/scripts-52717018-7386788331
, both volumes are mounted at /tmp-52717018-7386788331
which causes Kubernetes to reject the pod, as two volumes cannot be mounted to the same path.
Expected behavior
Given logs_base_dir
and scripts_base_dir
set to /tmp
,
Volumes should be mounted at /tmp/logs-{CI_PROJECT_ID}-{CI_JOB_ID}
, /tmp/scripts-{CI_PROJECT_ID}-{CI_JOB_ID}
Job pods should not be rejected by Kubernetes.
Relevant logs and/or screenshots
job log
[0KRunning with gitlab-runner 17.2.0 (6428c288)[0;m
[0K on gitlab-runner-7ff77d5df5-grs2t 2sm93yd4w, system ID: r_TSwzr5Hiht4J[0;m
[0K feature flags: FF_USE_ADVANCED_POD_SPEC_CONFIGURATION:true[0;m
section_start:1721422207:prepare_executor
[0K[0K[36;1mPreparing the "kubernetes" executor[0;m[0;m
[0KUsing Kubernetes namespace: gitlab-runner[0;m
[0KUsing Kubernetes executor with image quay.io/helmpack/chart-testing:latest ...[0;m
[0KUsing attach strategy to execute scripts...[0;m
section_end:1721422207:prepare_executor
[0Ksection_start:1721422207:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
[0KUsing FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...[0;m
[0;33mWARNING: Advanced Pod Spec configuration enabled, merging the provided PodSpec to the generated one. This is a beta feature and is subject to change. Feedback is collected in this issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29659 ...[0;m
section_end:1721422207:prepare_script
[0K[31;1mERROR: Error cleaning up pod: resource name may not be empty[0;m
[31;1mERROR: Job failed (system failure): prepare environment: setting up build pod: Pod "runner-2sm93yd4w-project-52717018-concurrent-0-x93l2sxa" is invalid: [spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp-52717018-7386788331": must be unique, spec.containers[1].volumeMounts[1].mountPath: Invalid value: "/tmp-52717018-7386788331": must be unique, spec.initContainers[0].volumeMounts[1].mountPath: Invalid value: "/tmp-52717018-7386788331": must be unique]. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information[0;m
Environment description
Kubernetes executor, running on vanilla Kubernetes 1.30.2.
config.toml contents
[[runners]]
environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"]
builds_dir = "/tmp/builds"
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
logs_base_dir = "/tmp"
scripts_base_dir = "/tmp"
privileged = false
allowPrivilegeEscalation = false
[[runners.kubernetes.pod_spec]]
name = "hostUsers"
patch = '''
[{"op": "add", "path": "/hostUsers", "value": false}]
'''
patch_type = "json"
Used GitLab Runner version
Running with gitlab-runner 17.2.0 (6428c288)
on gitlab-runner-7ff77d5df5-grs2t 2sm93yd4w, system ID: r_TSwzr5Hiht4J
feature flags: FF_USE_ADVANCED_POD_SPEC_CONFIGURATION:true
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image quay.io/helmpack/chart-testing:latest ...
Using attach strategy to execute scripts...
Possible fixes
As a workaround, users can set logs_base_dir
and scripts_base_dir
to different paths, but this doesn't address documentation or fix the config options to follow their intended functionality.
Additionally, I've opened !4893 (merged) to fix the config options to function as they were intended by !4805 (merged)