Add instrumentation for read_multi notifications
What does this MR do and why?
This MR instruments read_multi
and fetch_multi
methods in ActiveSupport::Cache::RedisCacheStore
.
Currently we only do not notify for cache_read_multi.active_support
(see how key is formed: https://github.com/rails/rails/blob/v6.1.6.1/activesupport/lib/active_support/cache.rb#L726) as Gitlab::Metrics::Subscribers::RailsCache
lacks it. In fairness, the Rails documentation does not include cache_read_multi.active_support
.
See gitlab-com/gl-infra/scalability#2004 (closed)
Note that this MR implements 2 of 3 points here !104335 (comment 1177638034). Cross-slot detection should be performed at a lower level (redis lib) since keys in the ActiveSupport::Cache stage are not normalized (https://github.com/rails/rails/blob/dc1242fd5a4d91e63846ab552a07e19ebf8716ac/activesupport/lib/active_support/cache.rb#L655)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
Local validation is trick since RackMiddleware is omitted in dev and test environments (https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/zz_metrics.rb#L17). I tested by making small modifications below:
Click to expand
diff --git a/app/services/projects/batch_count_service.rb b/app/services/projects/batch_count_service.rb
index 455c7211ab26..c955f97e8ff9 100644
--- a/app/services/projects/batch_count_service.rb
+++ b/app/services/projects/batch_count_service.rb
@@ -14,6 +14,7 @@ def refresh_cache_and_retrieve_data
services_by_cache_key = count_services.index_by(&:cache_key)
results = Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
+ Rails.cache.read_multi('a', 'b', 'c', 'd')
Rails.cache.fetch_multi(*services_by_cache_key.keys) do |key|
service = services_by_cache_key[key]
diff --git a/config/initializers/zz_metrics.rb b/config/initializers/zz_metrics.rb
index ff3ae9a24676..222ef5bbf954 100644
--- a/config/initializers/zz_metrics.rb
+++ b/config/initializers/zz_metrics.rb
@@ -14,7 +14,7 @@
# In development mode, we turn off eager loading when we're running
# `rails generate migration` because eager loading short-circuits the
# loading of our custom migration templates.
-if Gitlab::Metrics.enabled? && !Rails.env.test? && !(Rails.env.development? && defined?(Rails::Generators))
+if Gitlab::Metrics.enabled? #&& !Rails.env.test? && !(Rails.env.development? && defined?(Rails::Generators))
require 'pathname'
require 'connection_pool'
- Enable web_exporter metrics in
config/gitlab.yml
monitoring:
web_exporter:
enabled: true
-
Open up
localhost:3000
and open projects (these should triggerrefresh_cache_and_retrieve_data
function) -
Grep metrics to observe
read_multi
stats
curl localhost:8083/metrics 2> /dev/null | grep gitlab_cache_read_multikey_count_bucket
# HELP gitlab_cache_read_multikey_count Multiprocess metric
# TYPE gitlab_cache_read_multikey_count histogram
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="+Inf"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="1"} 0
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="10"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="100"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="1000"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="5"} 8
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="50"} 16
➜ gitlab git:(schin1-rails-cache-multikey-metrics) ✗ curl localhost:8083/metrics 2> /dev/null | grep gitlab_cache_operation_duration_seconds_bucket | grep multi
gitlab_cache_operation_duration_seconds_bucket{le="+Inf",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="0.0001",operation="read_multi"} 0
gitlab_cache_operation_duration_seconds_bucket{le="0.001",operation="read_multi"} 0
gitlab_cache_operation_duration_seconds_bucket{le="0.01",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="0.1",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="1.0",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="1.0e-05",operation="read_multi"} 0
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.