Improve trace masking performance
What does this MR do?
Reintroduces the trace improvements that were merged in !2677 (merged) but subsequently reverted due to problems with the masking setup.
What's the best way to test this MR?
- I'm initially pushing this MR without the fix that required the revert. We've since added multiple unit and integration tests that will fail the pipeline.
- The second commit to this MR will fix these issues and the pipeline should then succeed.
- https://gitlab.com/gitlab-org/gitlab-runner/-/pipelines/287251091 succeeded.
- Worth noting that the fix doesn't require additional tests as they were already added to specifically catch this as part of our corrective actions to the regression.
- For a manual QA:
-
Add a secret with any key, but with the value SECRET
-
Configure a
.gitlab-ci.yml
with:job: script: | for ((i = 0 ; i <= 3000 ; i++)); do num=$((1+$RANDOM % 5000)) text=$(printf "%${num}s\n" | tr " " "=") p=$(($RANDOM % $num)) echo ${i}. ${text:0:p}__SECRET__${num}${text:p} done
This script will print 1000 lines of text with the text
_SECRET_
inside at random locations on random length lines.It will also likely print far too much data and the data will cut off at a certain limit, also testing that that works as expected.
-
Open complete raw job log after the job finishes, and search for
_SECRET_
it should return zero results.
-
What are the relevant issue numbers?
closes #16705 (closed)
https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12574
Edited by Arran Walker