Gitlab runner for kubernetes only reads imagepullsecrets during runner registration, not on pod /deployment restart and not on new pipeline runs
Overview
The runner only loads the imagepullsecrets during the registration process, after that, if you change the user password and update the imagepullsecrets, it is unable to pull images. No matter what, you can restart the deployment or kill the runner pods and it won't read the new values from the imagepullsecrets.
- The ideal solution would be that the runner reads the imagepull secrets on every new pipeline run.
- If not possible, it should refresh its value on deployment restart or when killing the runner pod.
- The only working solution is to manually unregister the token, upgrade helm chart with the new token, and register again.
Is this behaviour documented? I think that there are lots of issues related to this like #27664 (closed)
Proposal for bug resolution
-
Documentation fix only: If you were using past versions of the Gitlab helm chart, now you don't need to specify the runners.imagePullSecrets, it is directly defined in the config.toml options of the runner, under config.runners.kubernetes. It is now specified in the docs in the very last sentence. This change was made in the chart to avoid problems caching credentials for longs periods of time, as in kubernetes if you don't change the secret name, the value is not read again until the pod restarts, also might happen issues with serviceaccount token cache expiration. You must now add the image_pull_secrets in the runner config directly as follows: