kubernetes executor uses wrong pull policies for services defined in config.toml

Summary

The docs state:

Use the pull_policy parameter in the config.toml file to specify a single or multiple pull policies. The policy controls how an image is fetched and updated, and applies to the build image, helper image, and any services.

However, since !4854 (merged) this is not true; services that were only defined in the config.toml default to pull policies [""] rather than what is configured in pull_policy, which then defaults to the cluster's default pull policy.

Steps to reproduce

.gitlab-ci.yml

stages:
  - test

Test:
  stage: test
  image: debian
  script: sleep 10m
  services:
    - name: ubuntu
      command: [ "sleep", "Infinity" ]

config.toml

listen_address = ":9252"
concurrent = 3
check_interval = 1
log_level = "debug"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "dm"
  limit = 3
  url = "https://gitlab.com/"
  id = 0
  token = "glrt-NOPENOPENOPE"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "kubernetes"
  shell = "bash"

  [runners.kubernetes]
    image = "ubuntu:22.04"

    poll_timeout = 60
    pull_policy = ["never"]

    [[runners.kubernetes.services]]
      name = "nginx"

  [runners.feature_flags]
    FF_USE_ADVANCED_POD_SPEC_CONFIGURATION = true
    FF_USE_POD_ACTIVE_DEADLINE_SECONDS = true
    FF_PRINT_POD_EVENTS = true
    FF_USE_FASTZIP = true

• ensure the nginx image is not available on the nodes
• ensure the build & helper images are on the nodes, e.g.

  for i in debian registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest  ; do kind load docker-image $i ; done

• trigger a build

Actual behavior

• it does no pull the build & helper image by inspecting the logs: ✅

    [...]
  Normal   Pulled   Container image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest" already present on machine
    [...]
  Normal   Pulled   Container image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest" already present on machine
    [...]

• it does not pull the service image defined in .gitlab-ci.yaml: ✅

    [...]
  Warning   ErrImageNeverPull   Container image "ubuntu" is not present with pull policy of Never
  Warning   Failed   Error: ErrImageNeverPull
    [...]

• it still pulls the nginx image, despite having the global pull policy of never ❌

    [...]
  Normal   Pulled   Successfully pulled image "nginx" in 9.233s (9.233s including waiting). Image size: 71027698 bytes.
    [...]

• the pod's containers have correct pull policies set, except the service container defined in the config.toml ❌

  : k get po -o yaml runner-xxx-xxx-xxx | yq '.spec.containers[] | {name, image, imagePullPolicy}'
  {
    "name": "build",
    "image": "debian",
    "imagePullPolicy": "Never"   # <- correct
  }
  {
    "name": "helper",
    "image": "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest",
    "imagePullPolicy": "Never"   # <- correct
  }
  {
    "name": "svc-0",
    "image": "nginx",
    "imagePullPolicy": "Always"   # <- WRONG
  }
  {
    "name": "svc-1",
    "image": "ubuntu",
    "imagePullPolicy": "Never"   # <- correct
  }

Expected behavior

• the service configured in config.toml should inherit the global pull policy
• thus the container should not be pulled
• in other words: services configured in config.toml should show the same behavior as services configured in the gitlb-ci.yaml

Relevant logs and/or screenshots

Environment description

n/a

Used GitLab Runner version

d8b6c0e7

Possible fixes

Ensure the runner gahters all services, also from config.toml, before populating the pull manager. Also ensure the pull policies for the services are defaulted correctly, all the way down to the global pull policies, before populating the pull manager or inside the pull manager.