Add metrics to track load_balancing for Sidekiq workers
Since we allowed Sidekiq workers to use load balancer capabilities we should introduce some metrics.
We should:
- Introduce the Prometheus metrics that will be used to keep track of how many times we queried
primary write log location
/replication log location
on the request level. - Add ::Gitlab::Metrics::Subscribers::ActiveRecord::DB_LOAD_BALANCING_COUNTERS to the structured log
- Introduces new Prometheus metric in Sidekiq
ServerMetrics
Middleware that will keep track of executed worker that were using load balancing capabilities, and it would provide information about worker data consistency, which database was chosen (primary or replica), if the job was retried because the replica was not ready, or we have fallen back to the primary.
Edited by Nikola Milojevic