Support alerts for Cluster Monitoring
Problem to solve
Currently we display cluster metrics to users, but we don't support alerts on them.
This can be a problem if a cluster starts to exceed its resources or requests, as no one knows unless they look at the page.
Intended users
Further details
Proposal
To achieve this, we should do two things:
- Support setting an alert which analyzes multiple queries. For example:
(Requests) / (Capacity) > 0.95
. - Collect usage ping data when a new alert is setup
- Allow alerts to be configured on Cluster Monitor screen
- Enable this alert rule by default (if possible)
Roadmap
- Move Cluster Metrics to Prometheus Endpoint https://gitlab.com/gitlab-org/gitlab-ce/issues/63044
- Move queries in cluster_metrics.yml into the DB, using an importer similar to
common_metrics_importer
. - Create a dashboard for cluster metrics and add it to the dashboards folder.
- Change the FE to start using the dashboard API instead of the existing API which reads from the cluster_metrics.yml file.
- Remove existing API.
- Move queries in cluster_metrics.yml into the DB, using an importer similar to
- Add alerting support for the new version of cluster metrics https://gitlab.com/gitlab-org/gitlab-ee/issues/12256
- Add support for alerting on multiple queries https://gitlab.com/gitlab-org/gitlab-ee/issues/12258
Design
Implementing this would mean introducing the alert buttons from the current metrics page onto the two charts on the cluster health page, as well as updating the alerts form. Proposal is to replace the first dropdown in the 'add alerts' dropdown with a field so users can input multiple queries if/as needed:
- Recommend adding in a tooltip to give users a little more help formatting the query since we are losing the dropdown:
- After a user has added an alert, alerts can be listed within a badge that would serve as a link to the alert edit/delete functionality:
Permissions and Security
Documentation
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
Edited by Amelia Bauerly