Errors connecting to Docker socket
Summary
I am receiving errors connecting to the Docker socket during the start of builds, the retries sometimes pass but a good percentage of the time the builds fail. Common error messages are:
ERROR: Preparation failed: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.18/containers/create?name=runner-c23dc1dc-project-35-concurrent-3-build: EOF
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed (system failure): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
The /var/run/docker.sock
exists on the host and Docker is running correctly, I can pull/run images as a normal user that is part of the docker
group.
Steps to reproduce
Set up a GitLab runner on Arch Linux
Actual behaviour
Get error messages about not being able to connect to Docker socket or HTTP Docker endpoint
Expected behaviour
A clean build
Relevant logs and/or screenshots
Running with gitlab-ci-multi-runner dev (HEAD)
on canny (c23dc1dc)
Using Docker executor with image vcatechnology/linux-mint-toolchain:17.3 ...
Using docker image sha256:65009e2043f704d40a314ca7a2f71d056880dddf55f719eba5f5fc3f2b4094bb for predefined container...
Pulling docker image vcatechnology/linux-mint-toolchain:17.3 ...
Using docker image vcatechnology/linux-mint-toolchain:17.3 ID=sha256:978e2de0c80f6af5e0f100f18158e3805d03e1885dc54d8c134ebc8b51827ecf for build container...
ERROR: Preparation failed: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.18/containers/create?name=runner-c23dc1dc-project-35-concurrent-3-build: EOF
Will be retried in 3s ...
Using Docker executor with image vcatechnology/linux-mint-toolchain:17.3 ...
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Will be retried in 3s ...
Using Docker executor with image vcatechnology/linux-mint-toolchain:17.3 ...
Using docker image sha256:65009e2043f704d40a314ca7a2f71d056880dddf55f719eba5f5fc3f2b4094bb for predefined container...
Pulling docker image vcatechnology/linux-mint-toolchain:17.3 ...
Using docker image vcatechnology/linux-mint-toolchain:17.3 ID=sha256:978e2de0c80f6af5e0f100f18158e3805d03e1885dc54d8c134ebc8b51827ecf for build container...
Running on runner-c23dc1dc-project-35-concurrent-3 via canny...
Cloning repository...
Cloning into '/mnt/builds/tool-chain/tool-chain'...
Checking out 1e171267 as mr-cache-third-party-refactor...
Skipping Git submodules setup
Environment description
Custom installation, using Arch Linux with pacman
provided docker
and gitlab-runner
.
[matt-clarkson@canny ~]$ docker info
Containers: 42
Running: 1
Paused: 0
Stopped: 41
Images: 9
Server Version: 17.04.0-ce
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 27.7GB
Data Space Total: 254.5GB
Data Space Available: 226.8GB
Metadata Space Used: 6.095MB
Metadata Space Total: 2.676GB
Metadata Space Available: 2.67GB
Thin Pool Minimum Free Space: 25.45GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.139 (2017-04-13)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary:
containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.10.11-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 47.16GiB
Name: canny
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Used GitLab Runner version
Version: dev
Git revision: HEAD
Git branch: HEAD
GO version: go1.8.1
Built: Thu, 27 Apr 2017 11:48:06 +0100
OS/Arch: linux/amd64
This is actually incorrect though - it's using v9.1.0 tag (at the time of submission) as seen in the PKGBUILD
]
concurrent = 8
check_interval = 0
[[runners]]
name = "canny"
url = "https://gitlab.xyz.com/"
token = "abcdef0123457890abcdef01234567"
executor = "docker"
builds_dir = "/mnt/builds"
cache_dir = "/mnt/cache"
[runners.docker]
tls_verify = false
image = "vcatechnology/arch-ci"
privileged = false
cap_add = ["SYS_PTRACE"]
disable_cache = false
extra_hosts = ["prewitt:192.168.4.1", "sobel:192.168.1.250"]
volumes = [
"/mnt/gitlab-runner/cache:/mnt/cache:rw",
"/mnt/gitlab-runner/third-party-prebuilt:/mnt/third-party-prebuilt:rw"
]
shm_size = 0
[runners.cache]