Skip to content

Send `SIGTERM` then `SIGKILL` to process in Shell executor

Steve Xuereb - Out of Office Back 2025-01-03 requested to merge shell-executor-use-process-pkg into master

👋 Target branch for now is add-process-group-to-process-pkg until !1743 (merged) is merged 👋

What does this MR do?

When the job is canceled first send SIGTERM and then send SIGKILL after a specific timeout. This uses the same package that is used for the custom executor.

Why was this MR needed?

Looking at #3376 (closed) most processes end up hanging when being killed directly, especially if it has child processes. Implement a graceful shutdown termination for both Windows/Unix systems behind a feature flag FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL which is turned off by default (so the new graceful termination is turned on by default).

Testing

Unix

  1. Download and compile the following go applicaiton

  2. Configure shell executor

    config.toml 
    [[runners]]
  name = "steve-mbp-gitlab.local"
  url = "http://192.168.1.79:3000/"
  token = "xxxx"
  executor = "shell"

  3. Have the following .gitlab-ci.yml

    .gitlab-ci.yml 
    job:
 script:
 - /Users/steve/Code/gitlab.com/steveazz/siglistener/siglistener

  4. Run job

  5. Run ps -opid,pgid,command which will show all the Running processes, and it's process groups

    ps -opid,pgid,command 
    47098 47098 go run main.go run -c /Users/steve/Code/gitlab.com/gitlab-org/gitlab/gitlab-runner-config.toml
47146 47098 /var/folders/h4/45_ps9bn4jb6w51g1ngy_n5c0000gn/T/go-build653627740/b001/exe/main run -c /Users/steve/Code/gitlab.com/gitlab-org/gitlab/gitlab-runner-config.toml
47289 47289 bash --login                                                                                                                                                      <--- In it's own process group
47293 47289 bash --login
47294 47289 /Users/steve/Code/gitlab.com/steveazz/siglistener/siglistener                 
47295 47289 sleep 60
47296 47289 sleep 60
47297 47289 sleep 60
47298 47289 sleep 60
47299 47289 sleep 60
47300 47289 sleep 60
47301 47289 sleep 60
47302 47289 sleep 60
47303 47289 sleep 60
47304 47289 sleep 60

  6. Run tail -f /tmp/siglistener.log

  7. Cancel Job

  8. Look at the /tmp/siglistener.log file which will print

    logs 
    2020/01/28 11:34:20 Received the following signal: terminated
2020/01/28 11:34:21 tick
2020/01/28 11:34:22 tick
2020/01/28 11:34:23 tick

  9. Wait 10 miniutes and process and child processes are killed

Windows

  1. Download and compile the following go applicaiton

  2. Configure shell executor

    config.toml 
    [[runners]]
  name = "steve-mbp-gitlab.local"
  url = "http://192.168.1.79:3000/"
  token = "xxxx"
  executor = "shell"

  3. Have the following .gitlab-ci.yml

    .gitlab-ci.yml 
    job:
  script:
  - C:\GitLab-Runner\builds\siglistener.exe

  4. Run job

  5. Run gwmi win32_process | % { "$($_.ProcessID) $($_.ParentProcessID) $($_.GetOwner().User) $($_.CommandLine)" } which will print all the process

    process 
    940 1904 Administrator "C:\GitLab-Runner\out\binaries\gitlab-runner-windows-amd64.exe" run -c .\config.toml
1476 940 Administrator powershell -noprofile -noninteractive -executionpolicy Bypass -command C:\Users\ADMINI~1\AppData\Local\Temp\2\build_script155394518\script.ps1
2892 1476 Administrator "C:\GitLab-Runner\builds\siglistener.exe"

  6. Cancel job

  7. Run gwmi win32_process | % { "$($_.ProcessID) $($_.ParentProcessID) $($_.GetOwner().User) $($_.CommandLine)" } and notice how 1476 940 Administrator powershell -noprofile -noninteractive -executionpolicy Bypass -command C:\Users\ADMINI~1\AppData\Local\Temp\2\build_script155394518\script.ps1 is not there anymore but 2892 1476 Administrator "C:\GitLab-Runner\builds\siglistener.exe" is.

  8. Wait 10 miniutes and the process should be killed.

Are there points in the code the reviewer needs to double check?

👋 When reviewing this merge request take a look at !1551 (closed) 👋

Does this MR meet the acceptance criteria?

  • Documentation created/updated
  • Added tests for this feature/bug
  • In case of conflicts with master - branch was rebased

What are the relevant issue numbers?

dev log
Edited by Steve Xuereb - Out of Office Back 2025-01-03

Merge request reports