Unexpected signal during runtime execution for docker executors on Windows machine
Summary
Gitlab runner crashes using Docker executor on Windows when running tests. Very short tests (less then a minute) don't seem to let the Gitlab runner crash.
I've tried to use the Docker for Windows Edge version - same result I've tried older versions of the Gitlab-runner - same result I've tried to give Docker for Windows a factory reset - same result.
I'm happy to supply additional information, because I'm stuck on this. Not sure if the problem is the result of a configuration problem / the gitlab-runner or docker.
Steps to reproduce
Starting a test from Gitlab CE where I php lint a large project (for example). Every test longer then aproximately a minute will result in the "fatal error: unexpected signal during runtime execution" error.
Relevant logs and/or screenshots
Runner output
c:\Gitlab-Runner>gitlab-runner.exe run
Starting multi-runner from c:\Gitlab-Runner\config.toml ...[0;m builds[0;m=0
Configuration loaded [0;m builds[0;m=0
Metrics server disabled [0;m
Checking for jobs... received [0;m job[0;m=5327 repo_url[0;m=http://********/******.git runner[0;m=60a65857
[0;33mWARNING: Job failed: exit code 1 [0;m [0;33mjob[0;m=5327 [0;33mproject[0;m=3 [0;33mrunner[0;m=60a65857
Checking for jobs... received [0;m job[0;m=5329 repo_url[0;m=http://********/******.git runner[0;m=60a65857
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x40e835]
goroutine 16 [running]:
runtime.throw(0x176c196, 0x2a)
/usr/local/go/src/runtime/panic.go:596 +0x9c fp=0xc042265e08 sp=0xc042265de8
runtime.sigpanic()
/usr/local/go/src/runtime/signal_windows.go:155 +0x18b fp=0xc042265e38 sp=0xc042265e08
runtime.lock(0x746f623d656c79cc)
/usr/local/go/src/runtime/lock_sema.go:43 +0x65 fp=0xc042265e70 sp=0xc042265e38
runtime.chansend(0x14cb980, 0x746f623d656c7974, 0xc042265fb0, 0xc042265f01, 0x7d358a, 0xc042265f9c)
/usr/local/go/src/runtime/chan.go:176 +0xba fp=0xc042265f20 sp=0xc042265e70
runtime.chansend1(0x14cb980, 0x746f623d656c7974, 0xc042265fb0)
/usr/local/go/src/runtime/chan.go:113 +0x4d fp=0xc042265f60 sp=0xc042265f20
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.ioCompletionProcessor(0x344)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/file.go:131 +0xea fp=0xc042265fd8 sp=0xc042265f60
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc042265fe0 sp=0xc042265fd8
created by gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.initIo
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/file.go:55 +0x87
goroutine 1 [chan receive, 3 minutes]:
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/ayufan/golang-kardianos-service.(*windowsService).Run(0xc0423e0d20, 0xc0422933b0, 0xc042440160)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/ayufan/golang-kardianos-service/service_windows.go:273 +0x166
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).Execute(0xc042293300, 0xc042307080)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:531 +0x32c
gitlab.com/gitlab-org/gitlab-runner/common.(Commander).Execute-fm(0xc042307080)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/command.go:24 +0x40
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/urfave/cli.HandleAction(0x14d1000, 0xc04236de80, 0xc042307080, 0xc0421ab000, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/urfave/cli/app.go:492 +0x83
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/urfave/cli.Command.Run(0x173add3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1757bee, 0x18, 0x0, ...)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/urfave/cli/command.go:210 +0xb75
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/urfave/cli.(*App).Run(0xc0423bb040, 0xc04203e3e0, 0x2, 0x2, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/urfave/cli/app.go:255 +0x776
main.main()
/go/src/gitlab.com/gitlab-org/gitlab-runner/main.go:58 +0x2b9
goroutine 18 [syscall, 3 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x10b
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x29
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x48
goroutine 21 [chan receive, 3 minutes]:
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/getsentry/raven-go.(*Client).worker(0xc0420748f0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/getsentry/raven-go/client.go:418 +0x72
created by gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/getsentry/raven-go.newClient
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/getsentry/raven-go/client.go:307 +0x141
goroutine 22 [chan receive]:
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x32c0e20)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/golang/glog/glog.go:882 +0x81
created by gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/golang/glog.init.1
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/golang/glog/glog.go:410 +0x224
goroutine 8 [select]:
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).updateConfig(0xc042293300, 0xc0421a7f18, 0xc0421ab680)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:319 +0x411
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).Run(0xc042293300)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:423 +0x2bf
created by gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).Start
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:284 +0x30c
goroutine 9 [chan send]:
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).feedRunner(0xc042293300, 0xc042307340, 0xc0421ab620)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:77 +0xc3
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).feedRunners(0xc042293300, 0xc0421ab620)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:95 +0x14d
created by gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).Run
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:406 +0x94
goroutine 10 [chan receive, 3 minutes]:
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).startWorkers(0xc042293300, 0xc0421ab680, 0xc0421ab740, 0xc0421ab620)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:198 +0x66
created by gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).Run
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:413 +0x260
goroutine 11 [select]:
gitlab.com/gitlab-org/gitlab-runner/common.(*Build).run(0xc042404dc0, 0x3277140, 0xc0423b6ba0, 0x327acc0, 0xc042088000, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:290 +0x667
gitlab.com/gitlab-org/gitlab-runner/common.(*Build).Run(0xc042404dc0, 0xc042052e80, 0x3278740, 0xc04229c280, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:398 +0x671
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processRunner(0xc042293300, 0x0, 0xc042307340, 0xc0421ab620, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:175 +0x7ca
gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processRunners(0xc042293300, 0x0, 0xc0421ab740, 0xc0421ab620)
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:183 +0x218
created by gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).startWorkers
/go/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go:199 +0xa6
goroutine 179 [chan receive]:
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc04235fa90, 0xc0423e1e30, 0x0, 0x0, 0x0, 0xc000000000, 0x326a7c0, 0x32606a8, 0xc042016600, 0xc0420173e0, ...)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/file.go:165 +0x24d
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc04235fa90, 0xc0421ea000, 0x8009, 0x8009, 0x42caf6, 0x29d3700, 0xc0421e1e00)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/file.go:188 +0xf1
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc0423a3040, 0xc0421ea000, 0x8009, 0x8009, 0xc042418250, 0xc042418258, 0xc042418250)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/pipe.go:114 +0x63
bufio.(*Reader).Read(0xc0423ec120, 0xc0421ea000, 0x8009, 0x8009, 0x91, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:199 +0x19f
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/docker/docker/pkg/stdcopy.StdCopy(0x5383b10, 0xc04229c280, 0x5383b10, 0xc04229c280, 0x3262240, 0xc0423ec120, 0xc0420417c0, 0x40, 0x2065756c61762064)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/docker/docker/pkg/stdcopy/stdcopy.go:105 +0xdc
gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*executor).watchContainer.func1(0xc042088000, 0xc0423a2fe0, 0xc0423ec8a0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker.go:898 +0xdf
created by gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*executor).watchContainer
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker.go:902 +0x6ba
goroutine 30 [IO wait]:
net.runtime_pollWait(0x3516670, 0x72, 0x32606a8)
/usr/local/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0423dc1a0, 0x72, 0x32606a8, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc0420500d8, 0xc0423dc060, 0x173ebbb, 0x7, 0x29d2d88, 0x1, 0x0, 0x0)
/usr/local/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc0423dc000, 0xc04248b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0420500e8, 0xc04248b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:181 +0x77
net/http.(*persistConn).Read(0xc04222e360, 0xc04248b000, 0x1000, 0x1000, 0x4, 0x454bf0, 0xc0421ab860)
/usr/local/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0423ec900)
/usr/local/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0423ec900, 0x1, 0xc0421a1be5, 0x1, 0x0, 0xc0421ab380, 0x0)
/usr/local/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc04222e360)
/usr/local/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1117 +0xa3c
goroutine 31 [select]:
net/http.(*persistConn).writeLoop(0xc04222e360)
/usr/local/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1118 +0xa61
goroutine 128 [semacquire]:
sync.runtime_notifyListWait(0xc042418280, 0xc000000011)
/usr/local/go/src/runtime/sema.go:298 +0x119
sync.(*Cond).Wait(0xc042418270)
/usr/local/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc042418240, 0xc042175000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc042050048, 0xc042175000, 0x1000, 0x1000, 0x91, 0x0, 0x0)
/usr/local/go/src/io/pipe.go:130 +0x53
bufio.(*Reader).fill(0xc04210bf10)
/usr/local/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadRune(0xc04210bf10, 0xa, 0x400000, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:270 +0xbe
gitlab.com/gitlab-org/gitlab-runner/network.(*clientJobTrace).process(0xc04229c280, 0xc042050048)
/go/src/gitlab.com/gitlab-org/gitlab-runner/network/trace.go:166 +0x105
created by gitlab.com/gitlab-org/gitlab-runner/network.(*clientJobTrace).start
/go/src/gitlab.com/gitlab-org/gitlab-runner/network/trace.go:118 +0x18e
goroutine 181 [sleep]:
time.Sleep(0x3b9aca00)
/usr/local/go/src/runtime/time.go:59 +0x107
gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*executor).waitForContainer(0xc042088000, 0xc04200ae00, 0x40, 0x0, 0xc04229c280)
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker.go:858 +0x297
gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*executor).watchContainer.func3(0xc0423ec960, 0xc042088000, 0xc04200ae00, 0x40)
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker.go:915 +0x46
created by gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*executor).watchContainer
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker.go:916 +0x76a
goroutine 109 [chan receive]:
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc0421fe0f0, 0xc042494000, 0x0, 0x0, 0x0, 0xc000000000, 0x326a7c0, 0x32606a8, 0x0, 0xc04250ffc0, ...)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/file.go:165 +0x24d
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc0421fe0f0, 0xc0423f7000, 0x1000, 0x1000, 0xc04250fa48, 0x456ee0, 0xc0423bbd40)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/file.go:188 +0xf1
gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc042210380, 0xc0423f7000, 0x1000, 0x1000, 0xc0421aa000, 0xc0423c3860, 0xc04250fbe5)
/go/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/Microsoft/go-winio/pipe.go:114 +0x63
net/http.(*persistConn).Read(0xc0422a2000, 0xc0423f7000, 0x1000, 0x1000, 0x4, 0x454bf0, 0xc0421aa000)
/usr/local/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0422de4e0)
/usr/local/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0422de4e0, 0x1, 0xc04250fbe5, 0x1, 0x0, 0xc04240e240, 0x0)
/usr/local/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc0422a2000)
/usr/local/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1117 +0xa3c
goroutine 145 [select]:
gitlab.com/gitlab-org/gitlab-runner/network.(*clientJobTrace).watch(0xc04229c280)
/go/src/gitlab.com/gitlab-org/gitlab-runner/network/trace.go:276 +0x18d
created by gitlab.com/gitlab-org/gitlab-runner/network.(*clientJobTrace).start
/go/src/gitlab.com/gitlab-org/gitlab-runner/network/trace.go:119 +0x1b0
goroutine 110 [select]:
net/http.(*persistConn).writeLoop(0xc0422a2000)
/usr/local/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1118 +0xa61
goroutine 149 [select]:
gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*executor).watchContainer(0xc042088000, 0x53412e8, 0xc04200b140, 0xc04200ae00, 0x40, 0x32622c0, 0xc0420db110, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker.go:918 +0xd1d
gitlab.com/gitlab-org/gitlab-runner/executors/docker.(*commandExecutor).Run(0xc042088000, 0xc04251c600, 0x1298, 0x0, 0x32770c0, 0xc04200b140, 0xc042531d60, 0xc042055ae0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/executors/docker/executor_docker_command.go:65 +0x2f4
gitlab.com/gitlab-org/gitlab-runner/common.(*Build).executeStage.func1(0xc000000018, 0x29ce198)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:179 +0x5b
gitlab.com/gitlab-org/gitlab-runner/helpers.(*BuildSection).Execute(0xc042531e48, 0x3262d00, 0xc042405040, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/helpers/build_section.go:48 +0xa4
gitlab.com/gitlab-org/gitlab-runner/common.(*Build).executeStage(0xc042404dc0, 0x32770c0, 0xc04200b140, 0x17452cd, 0xc, 0x327acc0, 0xc042088000, 0x1, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:181 +0x2e6
gitlab.com/gitlab-org/gitlab-runner/common.(*Build).executeScript(0xc042404dc0, 0x32770c0, 0xc04200b140, 0x327acc0, 0xc042088000, 0x0, 0x0)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:215 +0x242
gitlab.com/gitlab-org/gitlab-runner/common.(*Build).run.func1(0xc0424ec660, 0xc042404dc0, 0x32770c0, 0xc04200b140, 0x327acc0, 0xc042088000)
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:285 +0x61
created by gitlab.com/gitlab-org/gitlab-runner/common.(*Build).run
/go/src/gitlab.com/gitlab-org/gitlab-runner/common/build.go:286 +0x16e
Build output
Running with gitlab-runner 10.1.0 (c1ecf97f)
on PC-DEV (60a65857)
Using Docker executor with image php:5.6-apache ...
Using docker image sha256:f054df7bcf4af1254c5f7483e80c5152e35a2253c976e17af627f55b8b1ea00d for predefined container...
Pulling docker image php:5.6-apache ...
Using docker image php:5.6-apache ID=sha256:886842abb01efb966bb68f97d7f46c840aa6e74949928ffe611ce01ab86edd4c for build container...
Running on runner-60a65857-project-3-concurrent-0 via PC-JORIK...
Fetching changes...
Removing codeception.yml
Removing tests/_support/_generated/
Removing vendor/
HEAD is now at 0c37055 COMMIT
Checking out 0c370552 as BRANCH...
Skipping Git submodules setup
$ [ ! -d /cache/.composer/cache ] && mkdir -p /cache/.composer/cache
$ mkdir -p $HOME/.composer
$ ln -sf /cache/.composer/cache $HOME/.composer
$ find . -type f -not -path './vendor/*' -name "*.php" -print0 | xargs -i -0 -n1 -P2 php -l {} | grep -v "No syntax errors detected"
config.toml
concurrent = 1
check_interval = 0
[[runners]]
name = "PC-DEV"
url = "http://******/ci"
token = "********"
executor = "docker"
[runners.docker]
image = "docker:latest"
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
tls_verify = false
privileged = false
disable_cache = false
[runners.cache]
Environment description
-
Windows 10: Version 1703, build 15063.674)
-
Gitlab-runner: 10.1.0 (c1ecf97f)
-
Docker for Windows: stable, 17.09.0-ce-win33 (13620)
-
Docker image: php:5.6-apache
-
GitLab 10.0.4
-
GitLab Shell 5.9.0
-
GitLab Workhorse v3.0.0
-
GitLab API v4
Used GitLab Runner version
Version: 10.1.0
Git revision: c1ecf97f
Git branch: 10-1-stable
GO version: go1.8.3
Built: Sat, 21 Oct 2017 21:33:18 +0000
OS/Arch: windows/amd64