Capture helper service logs into job/tasks main trace log [docker]
Status
- MR approved
- merging blocked on completion of gitlab!100349 (merged)
What does this MR do?
This MR captures and streams logs from helper service containers to the CI task/job's main trace logs for the docker executor. This functionality is enabled when the variable
CI_DEBUG_SERVICES = 1
. While the logs are currently written to the jobs trace logs, they could easily be written elsewhere (file, log aggregator service, syslog...) in the future.
This approach relies on the docker.Client.CaptureLogs()
API; I also considered ContainerAttach()
as it was also suitable. In the end, for this use case, both API's are very similar; both return an FD in the form of an io.Reader()
into which the container's stdout
and stderr
are multiplexed, and on which stdcopy.StdCopy()
can be used to read the contents. CaptureLogs()
was a tad simpler so I chose it.
Why was this MR needed?
!2119 (merged) asks for service logs to be captured somehow. This is to help debug failing jobs when the failure is (at least in part) caused by behaviour in one of the service container services' (though not necessarily a failure to start said service container). A few possible approaches are mentioned in the issue; this MR takes the most "iteration friendly" (i.e. simplest) approach of copying the service logs inline into the main trace logs, but leaves room for the logs to be written elsewhere (e.g. a file) in the future.
What's the best way to test this MR?
-
BASELINE: Don't specify CI_DEBUG_SERVICES and run a CI job with a service container. The output of the main log trace should be unchanged from
main
. -
Set CI_DEBUG_SERVICES to a bogus value. The error message
invalid value '<xxx>' for CI_DEBUG_SERVICES variable
should appear in the main trace logs.
Example https://gitlab.com/avonbertoldi/test-project/-/jobs/2786325292
- Set
CI_DEBUG_SERVICES = 1
in the CI configuration- register a runner with a
docker
executor - create a job that includes a service which writes logs (example below)
- run the job (using a runner built from this branch)
- the service container's logs should appear in the job's main log trace in grey colour, with the container name prefixed to the log lines.
- register a runner with a
gitlab-ci.yaml
stages:
- test
variables:
POSTGRES_PASSWORD: password
CI_DEBUG_SERVICES: 1
format:
stage: test
image:
name: golang:latest
services:
- postgres:latest
script:
- go fmt $(go list ./... | grep -v /vendor/)
- go vet $(go list ./... | grep -v /vendor/)
- go test -race $(go list ./... | grep -v /vendor/)
Example https://gitlab.com/avonbertoldi/test-project/-/jobs/2841278059
What are the relevant issue numbers?
- #2119 (closed)
- GitLab runner - include logs for services (dock... (#28063 - closed)
-
gitlab#375180 (closed) is relevant only in that it's solutions block us from adding access control of job logs when this feature is enabled (as is done currently with
CI_DEBUG_TRACE
) as requested by @dcouture
Notes:
- Best reviewed commit-at-a-time.
Integration tests are still in the works and will be added to THIS MR when complete.- See !3564 (closed) for the
kubernetes
executor version of this MR.