Add total count metrics to Gitlab Internal Events
Problem
The current Internal Events implementation only supports metrics that count unique events per user (see #409993 (closed)). Redis metrics currently support the collection of all events of a specific kind (similar to a simple counter) for an all_time
timeframe.
To be able to deprecate the current methodology of collecting Redis metrics we need to provide the same functionality as part of the Internal Events Tracking, like we suggest in our prototypical documentation.
Desired Outcome
- We provide the ability to instrument metrics with
data_source: events
andtime_frame: all_time
that deliver a total_count - Instrumenting an event for such a metric via
Gitlab::InternalEvents.trackEvent
should lead to- a snowplow event for that metric being collected, including all provided data
- a Redis counter for that event being incremented
- Service Ping generation for this metric should lead to the current state of the Redis count being reported.
- Still reports the Service Ping context
- One of the currently already mirrored events for Snowplow and Redis is converted to the new
data_source
. For a list of potential events use:SELECT DISTINCT metrics_path FROM workspace_customer_success.wk_rpt_event_based_metric_counts_namespace_all_time
Potential Solution
Keep in mind that the service ping context for this kind of metric will be simplified as part of #411607 (comment 1394784802)
During #409993 (closed) new class is going to be created that will be responsible for loading metric definitions, caching it, and using that data to filter known events in internal events tracking, as well as to appoint attribute from event to be used for distinct counters via unique
attribute. That class in this issue would need to be extended to also load time frames, and use time_frame
along with event names and unique
to resolve which events should be tracked with which counters.
Additionally there is no general purpose Redis total counter class like HllRedisCounter
that allows for any event to be incremented. It would be required to create general purpose Redis total counter class, all shared logic between domain specific Redis total counter classes is already extracted to https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/base_counter.rb which should help in creation of general purpose Redis total counter class
Further Actions
TBD