Introduce filtering for total count metrics
What does this MR do and why?
Introduce filtering on additional properties for total count metrics
Total count metrics can now be defined with a filter which decides if an event will update a metric or not.
Here is an example of a metric definition using filters. The count of the metric will be increased if a perform_epics_action
event with a label
value of abc
is emitted or if a perform_issues_action
with a label
value of xyz
and a property
value of hello
:
key_path: counts.count_total_perform_epics_action_label_abc_or_label_xyz_monthly
description: 'Mountly count of some events. This description should be better'
product_section: dev
product_stage: plan
product_group: product_planning
performance_indicator_type: []
value_type: number
status: active
milestone: '17.0'
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/151646
time_frame: 28d
data_source: internal_events
data_category: optional
distribution:
- ee
tier:
- premium
- ultimate
events:
- name: perform_epics_action
filter:
label: abc
- name: perform_issues_action
filter:
label: xyz
property: hello
Since only additional properties can be used in filters, the only valid keys for a filter
are label
, property
and value
.
The main focus has been to support the migration of Redis metrics to internal events like decribed here: #424893 (closed)
Future work
This future work will continue immediately after this MR is merged.
- Documentation will be added in a separate MR when we migrate the first metrics
- Add filters for RedisHLL metrics
- Add filter support to the generator
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
- Create a few different metric definitions using the same event and different filters. You can get some like this
curl -s https://gitlab.com/gitlab-org/gitlab/-/commit/a6324efd6771bdac44654789178db064e1d39c03.diff | git apply -
- Start the monitor:
rails runner scripts/internal_events/monitor.rb perform_epics_action
- Start gdk and run the below commands one at a time and see the values in the monitor update as you expect.
Gitlab::InternalEvents.track_event('perform_epics_action', additional_properties: {label: 'abc'})
Gitlab::InternalEvents.track_event('perform_epics_action', additional_properties: {label: 'xyz'})
Gitlab::InternalEvents.track_event('perform_epics_action')
Gitlab::InternalEvents.track_event('perform_epics_action', additional_properties: {label: 'foo'})
Afterwards it should look like this:
Updated at 2024-05-02 07:55:36 UTC [PAUSED]
Monitored events: perform_epics_action
+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| RELEVANT METRICS |
+------------------------------------------------------------------------+----------------------+-----------------------+---------------+---------------+
| Key Path | Monitored Events | Instrumentation Class | Initial Value | Current Value |
+------------------------------------------------------------------------+----------------------+-----------------------+---------------+---------------+
| counts.count_total_perform_epics_action_label_abc_monthly | perform_epics_action | TotalCountMetric | 0 | 1 |
| counts.count_total_perform_epics_action_label_abc_or_label_xyz_monthly | perform_epics_action | TotalCountMetric | 0 | 2 |
| counts.count_total_perform_epics_action_label_xyz_monthly | perform_epics_action | TotalCountMetric | 0 | 1 |
| counts.count_total_perform_epics_action_monthly | perform_epics_action | TotalCountMetric | 0 | 4 |
+------------------------------------------------------------------------+----------------------+-----------------------+---------------+---------------+
+----------------------------------------------------------------------------------------------------------------------+
| SNOWPLOW EVENTS |
+----------------------+--------------------------+-----------------------+---------+--------------+------------+------+
| Event Name | Collector Timestamp | Category | user_id | namespace_id | project_id | plan |
+----------------------+--------------------------+-----------------------+---------+--------------+------------+------+
| perform_epics_action | 2024-05-02T07:55:32.657Z | InternalEventTracking | | | | |
| perform_epics_action | 2024-05-02T07:54:40.166Z | InternalEventTracking | | | | |
| perform_epics_action | 2024-05-02T07:53:41.407Z | InternalEventTracking | | | | |
| perform_epics_action | 2024-05-02T07:52:13.917Z | InternalEventTracking | | | | |
+----------------------+--------------------------+-----------------------+---------+--------------+------------+------+
Numbered steps to set up and validate the change are strongly suggested.
Related to #435338 (closed)