Skip to content

WIP: Live traces improvements

Grzegorz Bizon requested to merge fix/gb/improve-ci-trace-chunks into master

What does this MR do?

Live traces / incremental CI/CD traces improvements.

How does it work currently?

Sequence diagram of the most important sequences (incomplete):

sequenceDiagram
    autonumber
    participant R as Runner
    participant G as GitLab (rails)
    participant I as Redis
    participant D as Database
    participant O as Object store
    participant U as User
    loop partial trace
      Note right of R: Successful dataflow
	    R->>+G: trace [build.id, offset, data]
      G-->>+I: lock build trace [trace:write:lock:build.id]
      G->>+D: find or create chunk [chunk.index]
      D-->>-G: chunk [id, index]
      G-->>+I: lock chunk [trace_write:build.id:chunks:chunk.index]
      G->>I: append chunk data [chunk.index, data]
      I-->>-G: unlock chunk [trace_write:build.id:chunks:chunk.index]
      G->>D: truncate chunks [offset]
      G-->>+I: lock chunk [trace_write:build.id:chunks:chunk.index]
      G->>I: truncate chunk [chunk.index, offset]
      I-->>-G: unlock chunk [trace_write:build.id:chunks:chunk.index]
      I-->>-G: unlock build trace [trace:write:lock:build.id]
      G-->>-R: 200 ok
    end

    Note right of R: Retrieve trace
    U->>+G: GET build.id/trace
    loop every trace chunk
      G->>+D: find chunk [index]
      D-->>-G: chunk [id]
      G->>+I: read chunk data [chunk.index]
      I-->>-G: chunk data [data, size]
    end
    G-->>-U: build trace [json]
    
    Note right of R: Chunk overflow
    loop retry sending trace chunk in case of failure
      R->>+G: trace [build.id, offset, data]
      G-->>+I: lock build trace [trace:write:lock:build.id]
      G->>+D: find or create chunk [chunk.index]
      D-->>-G: chunk [id, index]
      G->>G: chunk overflow 128kb [exception]
      I-->>-G: unlock build trace [trace:write:lock:build.id]
      G-->>R: 500 internal server error
    end

    Note right of R: Chunk full
	  R->>+G: trace [build.id, offset, data]
    G-->>+I: lock build trace [trace:write:lock:build.id]
    G->>+D: find or create chunk [chunk.index]
    D-->>-G: chunk [id, index]
    G-->>+I: lock chunk [trace_write:build.id:chunks:chunk.index]
    G->>I: append chunk data [chunk.index, data]
    G->>G: chunk full [index]
    I-->>-G: unlock chunk [trace_write:build.id:chunks:chunk.index]
    G->>D: truncate chunks [offset]
    G-->>+I: lock chunk [trace_write:build.id:chunks:chunk.index]
    G->>I: truncate chunk [chunk.index, offset]
    I-->>-G: unlock chunk [trace_write:build.id:chunks:chunk.index]
    I-->>-G: unlock build trace [trace:write:lock:build.id]
    G-->>-R: 200 ok
    G-->>+I: lock build trace [trace:write:lock:build.id]
    G->>+I: read chunk data [chunk.index]
    I-->>-G: chunk data [data, size]
    G->>+D: update data store type [chunk.id]
    D-->>-G: data store type updated [chunk.id]
    G->>+O: send chunk data [data, size]
    O-->>-G: data written [chunk.index]
    G->>+I: delete chunk data [chunk.index]
    I-->>-G: chunk data removed [chunk.index]
    I-->>-G: unlock build trace [trace:write:lock:build.id]

    Note right of R: Archive traces sequence
    loop Archive Traces Cron - 17 * * * *
      loop every trace chunk
        G-->>+I: lock build trace [trace:write:lock:build.id]
        G->>+I: read chunk data [chunk.index]
        I-->>-G: chunk data [data, size]
        G->>+D: update data store type [chunk.id]
        D-->>-G: data store type updated [chunk.id]
        G->>+O: send chunk data [data, size]
        O-->>-G: data written [chunk.index]
        G->>+I: delete chunk data [chunk.index]
        I-->>-G: chunk data removed [chunk.index]
        I-->>-G: unlock build trace [trace:write:lock:build.id]
      end 
    end

Conformity

Issues

Closes #34781 (closed)

Edited by Grzegorz Bizon

Merge request reports