Kube attach strategy hangs when log file is deleted.
Summary
When running with FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY
set to false
and you delete the log file from disk the job ends up running until the job times out.
Steps to reproduce
.gitlab-ci.yml
variables:
DURATION: 3600
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: "false"
job:
script:
- 'for i in $(seq 1 $DURATION); do echo $(date); sleep 1; done'
- echo "done"
- Run a the above
.gitlab-ci.yml
file. - When the job log start to output the date-time run the follow command to delete the log file
kubectl exec -it -c helper $POD_RUNNING_JOB -- sh -c 'rm /builds/root/playground.tmp/logs/output.log'.
/builds/root/playground.tmp` should be the path where the build is running.
Actual behavior
We get an error that a file doesn't exist anymore and a timeout error and job runs forever.
Expected behavior
Gracefully fail the job
Relevant logs and/or screenshots
Environment description
config.toml contents
[[runners]]
name = "kubernetes"
url = "http://192.168.190.160:3000"
token = "o5hBR1gQYzi6gwMvb_d1"
executor = "kubernetes"
[runners.kubernetes]
image = "alpine:3.11"
privileged = true
namespace = "default"
Used GitLab Runner version
Master branch