AggregatedMetric property_name handling
What does this MR do and why?
Related to #415139 (closed)
We are already saving RedisHLL counters separately for each property name. Now, let's make it so that we also read those new counters for AggregatedMetric.
All AggregatedMetrics have an options.aggregate.attribute
option defined. This option correlates to what is called a property_name
inside the RedisHLLCounter
class. In this MR, we will be passing this option into the RedisHLLCounter
class, so that the counter class can use it for reading the data.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Enable the feature flag:
Feature.enable(:redis_hll_value_name_tracking)
A. Test AggregatedMetric
:
- Create a new metric or modify one that has just internal_events [for example: this one], making it so that it uses a new unique value for its
events.unique
property [in the given example: we can just changeuser_id
toproject_id
] - Trigger the metric's events a few times, sending the
events.unique
property's pre-dot part asproperty_name
andtime
set to last week [for example:Gitlab::UsageDataCounters::HLLRedisCounter.track_event('g_edit_by_sfe', values: SecureRandom.uuid, property_name: 'project', time: 1.week.ago)
] - Check the current value for the events, for example:
Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: %w[g_edit_by_web_ide g_edit_by_sfe g_edit_by_snippet_ide], end_date: Date.today, start_date: 3.week.ago, property_name: 'project.id')
- Generate a new Service Ping:
sp = Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)
- Check the metric's key path [example:
sp['redis_hll_counters']['ide_edit']['ide_edit_total_unique_counts_monthly']
]. It should have the same value as the one we saw using theHLLRedisCounter
class.
B. Make sure we haven't affected existing metrics
Something that we want to be fully sure about is that we're not affecting any metrics with this code. To check that, we can:
- Enable the gdk and perform some actions: for example, create an issue, add comments, create an epic, add an emote reaction
- Checkout the master branch and generate the next week's Service Ping in the rails console:
require 'active_support/testing/time_helpers'
include ActiveSupport::Testing::TimeHelpers
sp = travel_to(Date.today + 7.days) { Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) }
- Checkout this MR's branch and go back to the rails console:
reload!
sp2 = travel_to(Date.today + 7.days) { Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) }
- Select all the keys that are different between these two service pings:
def find_different_keys(h1, h2, result, current_key = '')
raise StandardError if h1.keys != h2.keys
diffs = h1.keys.each do |k|
if h1[k].is_a?(Hash)
result = find_different_keys(h1[k], h2[k], result, current_key + "#{k}.")
else
(h1[k] != h2[k]) ? result += [current_key + "#{k}"] : nil
end
end
result
end
find_different_keys(sp, sp2, [])
This should only list metrics that are not calculated using RedisHLL. For example, in my case, it was:
["elasticsearch_enabled",
"geo_enabled",
"gitlab_pages.enabled",
"database.flavor",
"object_store.artifacts.enabled",
"object_store.artifacts.object_store.enabled",
"object_store.artifacts.object_store.direct_upload",
"object_store.artifacts.object_store.background_upload",
"object_store.external_diffs.enabled",
"object_store.external_diffs.object_store.enabled",
"object_store.external_diffs.object_store.direct_upload",
"object_store.external_diffs.object_store.background_upload",
"object_store.lfs.enabled",
"object_store.lfs.object_store.enabled",
"object_store.lfs.object_store.direct_upload",
"object_store.lfs.object_store.background_upload",
"object_store.uploads.enabled",
"object_store.uploads.object_store.enabled",
"object_store.uploads.object_store.direct_upload",
"object_store.uploads.object_store.background_upload",
"object_store.packages.enabled",
"object_store.packages.object_store.enabled",
"object_store.packages.object_store.direct_upload",
"object_store.packages.object_store.background_upload",
"topology.duration_s"]
If you got any other keys, they can be checked by searching for the key in https://metrics.gitlab.com/ [for exmaple: here]. After loading a row, make sure that the source
column for it is not redis_hll
or internal_events
. For the given example, the source is system
.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.