Do not permit use of reserved log labels
In gitlab-org/gitlab!111285 (merged) we found that there is some surprising behavior in our logging stack: if the application logs events using "reserved" labels such as type
(which IIUC gets injected by fluentd after the event leaves the rails application), then we quietly accept these and they will be overwritten later on.
There are two problems here:
- Clients using generic label names such as
type
with context-specific semantics (in this caseCI artifact type
) - External services overwriting certain "reserved" labels
I think we should do one or both of:
- Not allowing user applied labels to appear at the top-level label "namespace", e.g. by always moving them under an
extra.*
prefix. This would have resolved the clash since there would have beentype
andextra.type
- Fail fast i.e. reject labels we know have existing meaning such as
type
. It would also help enforce better naming policies since in this example, I thinkci_artifact_type
would have been a much better label name to begin with.
PS: I am not sure this is solvable in labkit-ruby alone since we apply gitlab-rails specific logic and use many different loggers too.