Skip to content

WIP: Make trace chunks append to be performant

Kamil Trzciński (Back 2025-01-01) requested to merge make-trace-chunks-append-performant into master

What does this MR do?

Previously, we would always read all data before append. This is very inefficient, as it causes to transfer a big amount of data due to 128kB of trace chunks.

Now, since Redis our temporary storage, we try to use efficient append of traces, and try to avoid data transfer at all.

This makes us to move away from using a combination of:

  1. get key
  2. setex key "full-string" "expiry-time"

To use (if possible):

  1. strlen key,
  2. setrange offset "string-to-append",
  3. `expire key "expiry-time".

This means that we try to transfer only incremental changes to Redis, instead of transfering back and forth full chunks.

Redis log:

# NOW
1576588271.751543 [0 172.20.0.8:41282] "strlen" "gitlab:ci:trace:6180:chunks:0"
1576588271.752090 [0 172.20.0.8:41282] "multi"
1576588271.752155 [0 172.20.0.8:41282] "setrange" "gitlab:ci:trace:6180:chunks:0" "6" "me"
1576588271.752165 [0 172.20.0.8:41282] "expire" "gitlab:ci:trace:6180:chunks:0" "604800"
1576588271.752175 [0 172.20.0.8:41282] "exec"
# BEFORE
1576588411.698971 [0 172.20.0.8:41282] "get" "gitlab:ci:trace:6180:chunks:0"
1576588411.699198 [0 172.20.0.8:41282] "set" "gitlab:ci:trace:6180:chunks:0" "dtestmetestme" "EX" "604800"

Validation

I did not yet validate the handling of this method with binary strings, aka. UTF-8 content, if strlen does return byte length or utf-8 length.

Impact

It will effectively make Redis and our networking happy:

Does this MR meet the acceptance criteria?

Conformity

Resolves #34781 (closed)

Edited by Grzegorz Bizon

Merge request reports