Skip to content

`gitlab-runner start` throws "FATAL: Failed to start gitlab-runner: exit status 134" when started prior to being logged in

Summary

A customer might've discovered a previously unknown macOS quirk
I managed to reproduce the behavior on an AWS EC2 instance.

Something's preventing gitlab-runner start from doing its jobs before the user has logged in to a UI session, and it doesn't look LaunchAgent vs LaunchDaemon related this time around.

Steps to reproduce

  1. Spin up a fresh EC2 mac2.metal (arm64-mac in my case) instance
  2. Log in via SSH as ec2-user
  3. Install GitLab Runner per our docs
  4. Register a Runner
  5. Start the Runner

Ticket Id

https://gitlab.zendesk.com/agent/tickets/560157

Architecture

N/A

Current behaviour

gitlab-runner dies with SIGABRT:

ec2-user@host ~ % gitlab-runner start
Runtime platform                                    arch=arm64 os=darwin pid=815 revision=f5c15f76 version=16.11.3
FATAL: Failed to start gitlab-runner: exit status 134

If the machine is rebooted at this point, the error will change to the known and documented Input/output error.

Expected behaviour

Optimally

gitlab-runner starts

Realistically

gitlab-runner dies with the Input/output error

ec2-user@host ~ % gitlab-runner start                    
Runtime platform                                    arch=arm64 os=darwin pid=573 revision=f5c15f76 version=16.11.3
FATAL: Failed to start gitlab-runner: "launchctl" failed with stderr: Load failed: 5: Input/output error
Try running `launchctl bootstrap` as root for richer errors.

Relevant logs and/or screenshots

Runner console output 
    ┌───┬──┐   __|  __|_  )
    │ ╷╭╯╷ │   _|  (     /
    │  └╮  │  ___|\___|___|
    │ ╰─┼╯ │  Amazon EC2
    └───┴──┘  macOS Sonoma 14.6.1

ec2-user@host ~ % sudo mkdir -p /usr/local/bin
ec2-user@host ~ % sudo curl --output /usr/local/bin/gitlab-runner \
"https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/v16.11.3/binaries/gitlab-runner-darwin-arm64"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60.7M  100 60.7M    0     0  35.9M      0  0:00:01  0:00:01 --:--:-- 35.9M
ec2-user@host ~ % sudo chmod +x /usr/local/bin/gitlab-runner
ec2-user@host ~ % sudo su - ec2-user
ec2-user@host ~ % cd
ec2-user@host ~ % vim token
ec2-user@host ~ % source token 
ec2-user@host ~ % /usr/local/bin/gitlab-runner register \
--non-interactive \
--url "https://gitlab.example.org" \
--token "${RUNNER_TOKEN}" \
--executor "shell" \
--description "ZD-560157"
Runtime platform                                    arch=arm64 os=darwin pid=813 revision=f5c15f76 version=16.11.3
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Created missing unique system ID                    system_id=s_b403a8ea4605
Verifying runner... is valid                        runner=CthfUx_ry
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
 
Configuration (with the authentication token) was saved in "/Users/ec2-user/.gitlab-runner/config.toml" 
ec2-user@host ~ % gitlab-runner start
Runtime platform                                    arch=arm64 os=darwin pid=815 revision=f5c15f76 version=16.11.3
FATAL: Failed to start gitlab-runner: exit status 134

Environment description

config.toml contents 
concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ZD-560157"
  url = "https://REDACTED"
  id = 16
  token = "glrt-REDACTED"
  token_obtained_at = 2024-08-27T09:51:05Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Used GitLab Runner version

Version:      16.11.3
Git revision: f5c15f76
Git branch:   16-11-stable
GO version:   go1.21.9
Built:        2024-07-25T17:34:34+0000
OS/Arch:      darwin/arm64

Possible fixes

  • Start gitlab-runner with the --debug flag, for some reason
  • Do a UI login prior to starting the Runners, via VNC for example