Firing prometheus alerts configured by gitlab are not found by environment
Summary
Gitlab::Alerting::Alert#parse_gitlab_alert_from_payload
does not account for the environment on which the alert is configured. If there are multiple environments which have alerting rules configured for a given metric, the firing alert could be associated with the wrong one.
A prometheus_metric_id
and project
are not sufficient to uniquely identify a PrometheusAlert
.
Proposal
gitlab_environment_name
field
Now - Use the existing Gitlab::Alerting::Alert#parse_environment_from_payload
should be attempted as a part of finding the alerting rule in the first place.
gitlab_prometheus_alert_id
in the gitlab-configured payload for alerting rules
Future - Start including the actual We can configure with alerting rules one value which can be used to uniquely identify an alert, rather than all three of project
, metric_id
, and environment_name
.
Likewise, we should probably consider refactoring Projects::Prometheus::AlertsController
to rely on the actual prometheus_alert_id
rather than the combination of project
, environment
, and prometheus_metric_id
.