Add instrumentation for Redis connection pools
What does this MR do and why?
This MR adds instrumentation for connection pools (the main purpose is to observe Redis connection pools).
The 2 gauges gitlab_connection_pool_size
and gitlab_connection_pool_available_count
will help track the utilisation of connection pools. For sidekiq, the connection pool pool_name
will be "unknown" for Sidekiq 6.x.
In Sidekiq 7.x, pools are named:
- https://github.com/sidekiq/sidekiq/blob/v7.2.0/lib/sidekiq/capsule.rb#L94: capsule pools are named after the capsule
- https://github.com/sidekiq/sidekiq/blob/v7.2.0/lib/sidekiq/config.rb#L133: local pool is named "internal"
See related issue: gitlab-com/gl-infra/scalability#2564 (comment 1672791437). This closes gitlab-com/gl-infra/scalability#2649 (closed)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- In rails console, run a thread
Thread.new do
Gitlab::Redis::Queues.with do |r|
r
sleep(1000)
end
end
- Check the pool size:
require 'prometheus/client/formats/text.rb'
Prometheus::Client::Formats::Text.marshal_multiprocess.split("\n").filter{|x| x.include?("gitlab_connection_pool_available") && x.include?(Process.pid) }
"gitlab_connection_pool_available_count{pool_key=\"pool-177440\",pool_name=\"queues\",pid=\"process_45289\"} 4"]
- Checkout another connection and check that the available count is 3.
Gitlab::Redis::Queues.with { |c| c.ping }
Prometheus::Client::Formats::Text.marshal_multiprocess.split("\n").filter{|x| x.include?("gitlab_connection_pool_available") && x.include?(Process.pid) }
There should be a line like the following, since 2 connections are checked out. One is still held by the pool, and the gauge was set to 3
when Gitlab::Redis::Queues.with { |c| c.ping }
was run.
"gitlab_connection_pool_available_count{pool_key=\"pool-177440\",pool_name=\"queues\",pid=\"process_45289\"} 3"]
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.