Include worker_id in application logs
In !66694 (merged) we added a worker's process ID to application logs. This was done to support diagnosis during production incidents.
In &8105 I found that in addition to that, having the logical worker_id
would be more useful instead. This is because in Thanos, we do not collect PIDs, since that label value would be essentially unbounded. Plus, it's ephemeral; for instance, puma_0
might get restarted and hence run under a new PID, but it's still the same worker. So when breaking down by the pid
label in Thanos (this is actually the worker ID, not the process ID), we cannot currently correlate these data with logs.
I suggest we extend InstrumentationHelper to also log PidProvider#worker_id
. This is used by both Puma and Sidekiq, so it will work for both.
For Puma, we currently run 7 processes in SaaS: 1 master + 6 workers. So the label cardinality is 7.
For Sidekiq, we run a single process, so label cardinality is 1.