Handle log timestamps in Ansi2Json
What does this MR do and why?
This MR improves Gitlab::Ci::Ansi2json::Converter
so that it parses and outputs log timestamps generated by gitlab-runner
when the FF_TIMESTAMPS
feature flag is enabled.
This MR is probably easier to review commit-by-commit.
The processing of timestamps in this MR introduces a performance penalty of ~8% when compared with master
, but given that this is behind a gitlab-runner FF, it seems OK. I've created a follow-up issue to address the performance issues, before we roll this out more widely.
Logs are processed by the Gitlab::Ci::Ansi2html
and Gitlab::Ci::Ansi2json
classes, to produce HTML and JSON variants. Recently, the runner has introduced a FF_TIMESTAMPS
feature flag, which will cause it to emit lines with a slightly modified logic:
E.g. 2024-05-14T11:19:20.000000Z 00O+Hey there!
. The first 32 characters are a timestamp prefix, and must be treated as optional. If present, Ansi2json
must include each line's respective timestamp in its output.
Format:
<timestamp> <stream number><stream type><full line type>
-
timestamp
: UTC RFC3339 timestamp -
stream number
: 1 byte (2 hex chars) stream number -
stream type
:E
/O
(Err or Out) -
full line type
:+
if line is continuation of previous line,
Closes: #455581 (closed)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Job without FF_TIMESTAMPS (no change)
Before | After |
---|---|
Job with FF_TIMESTAMPS (timestamps removed)
Before | After |
---|---|
Performance benchmarks
runs = 100; Benchmark.bm do |x|
x.report('Large log with timestamps (this branch)') do
runs.times do
json = File.open("~/Downloads/raw-large.txt", "r") { |stream| converter = Gitlab::Ci::Ansi2json::Converter.new.convert(stream, nil) }
end
end
x.report('Large log without timestamps (this branch)') do
runs.times do
json = File.open("~/Downloads/raw-no-ts-large.txt", "r") { |stream| converter = Gitlab::Ci::Ansi2json::Converter.new.convert(stream, nil) }
end
end
end
Scenario | user | system | total | real |
---|---|---|---|---|
Large log without timestamps (master ) |
50.721283 | 0.442331 | 51.163614 | 51.914806 |
Large log with timestamps (master ) |
52.246681 | 0.502974 | 52.749655 | 53.976508 |
Large log without timestamps (this branch) | 52.938921 | 0.364169 | 53.303090 | 53.849536 |
Large log with timestamps (this branch) | 57.927132 | 0.381748 | 58.308880 | 58.697001 |
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.