Kubernetes Executor not assigning configured resource requests
Summary
When submitting jobs to the Kubernetes Executor, the resources
tag of the job is not filled with the values defined but remains empty instead.
This leads to k8s not scaling the resources and terminating runner jobs if they exceed the existing resources.
The build container is then configured by the Kubernetes Executor with the following config:
name: build
resources: {}
securityContext:
privileged: true
Confusingly, those resource request values are being recognized properly in rare occasions, leading to the desired
name: build
resources:
limits:
cpu: "4"
memory: 2Gi
requests:
cpu: 450m
memory: 256Mi
securityContext:
privileged: true
Couldn't identify a pattern that would cause this seemingly random behaviour.
Steps to reproduce
The Kubernetes Runner was deployed with Helm using
helm repo add gitlab https://charts.gitlab.io
helm init --upgrade
helm install \
--namespace gitlab-managed-apps \
--name GITLAB_USER_NAME \
--set runnerRegistrationToken=GITLAB_RUNNER_REGISTRATION_TOKEN \
-f gitlab-runner-setup.yaml \
gitlab/gitlab-runner
The corresponding gitlab-runner-setup.yaml
was:
gitlabUrl: https://gitlab.com/
rbac:
create: true
runners:
privileged: true
image: alpine-latest
builds:
cpuLimit: 4
memoryLimit: 1Gi
cpuRequests: 450m
memoryRequests: 256Mi
services:
cpuLimit: 100m
memoryLimit: 128Mi
cpuRequests: 30m
memoryRequests: 64Mi
helpers:
cpuLimit: 100m
memoryLimit: 128Mi
cpuRequests: 20m
memoryRequests: 64Mi
resources:
limits:
memory: 128Mi
cpu: 100m
requests:
memory: 32Mi
cpu: 10m
which leads to the proper Runner pod YAML. However, the environment variables don't seem to get converted when the Kubernetes Executor creates the job Pod configuration.
...
containers:
- command:
- /bin/bash
- /scripts/entrypoint
env:
- name: CI_SERVER_URL
value: https://gitlab.com/
- name: CLONE_URL
- name: RUNNER_REQUEST_CONCURRENCY
value: "1"
- name: RUNNER_EXECUTOR
value: kubernetes
- name: REGISTER_LOCKED
value: "true"
- name: RUNNER_TAG_LIST
- name: KUBERNETES_IMAGE
value: alpine-latest
- name: KUBERNETES_PRIVILEGED
value: "true"
- name: KUBERNETES_NAMESPACE
value: gitlab-managed-apps
- name: KUBERNETES_CPU_LIMIT
value: "4"
- name: KUBERNETES_MEMORY_LIMIT
value: 2Gi
- name: KUBERNETES_CPU_REQUEST
value: 450m
- name: KUBERNETES_MEMORY_REQUEST
value: 256Mi
- name: KUBERNETES_SERVICE_ACCOUNT
- name: KUBERNETES_SERVICE_CPU_LIMIT
value: 100m
- name: KUBERNETES_SERVICE_MEMORY_LIMIT
value: 128Mi
- name: KUBERNETES_SERVICE_CPU_REQUEST
value: 30m
- name: KUBERNETES_SERVICE_MEMORY_REQUEST
value: 64Mi
- name: KUBERNETES_HELPER_CPU_LIMIT
value: 100m
- name: KUBERNETES_HELPER_MEMORY_LIMIT
value: 128Mi
- name: KUBERNETES_HELPER_CPU_REQUEST
value: 20m
- name: KUBERNETES_HELPER_MEMORY_REQUEST
value: 64Mi
- name: KUBERNETES_HELPER_IMAGE
- name: KUBERNETES_PULL_POLICY
image: gitlab/gitlab-runner:alpine-v12.0.0
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /bin/bash
- /scripts/check-live
failureThreshold: 3
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: GITLAB_USER_NAME-gitlab-runner
ports:
- containerPort: 9252
name: metrics
protocol: TCP
readinessProbe:
exec:
command:
- /usr/bin/pgrep
- gitlab.*runner
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 10m
memory: 32Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /secrets
name: runner-secrets
- mountPath: /home/gitlab-runner/.gitlab-runner
name: etc-gitlab-runner
- mountPath: /scripts
name: scripts
dnsPolicy: ClusterFirst
...
.gitlab-ci.yml
image: "alpine:latest"
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
HELM_VERSION: 2.14.1
KUBERNETES_VERSION: 1.12.7
TZ: "Europe/Berlin"
stages:
- build-base
- test
- deployment
- cleanup
test-backend:
stage: test
image: registry.gitlab.com/<IMAGE_NAME>
services:
- postgres:9.6-alpine
script:
- cd products/backend
- pip install --quiet --user -r requirements-testing.txt
- make -f Makefile-k8s.mk lint
- make -f Makefile-k8s.mk test-cov
Used GitLab Runner version
Running with gitlab-runner 12.0.0 (6946bae7)
Using Kubernetes executor with image registry.gitlab.com/...