Allow generic endpoint to receive alerts from external Prometheus
Problem to solve
In order to setup Alert Management for a external Prometheus instance users have to complete the following steps:
- Enable Alert Management via Settings > Integration > Alerts endpoint
- Enable and configure manual Prometheus via Settings > Integrations > Prometheus
- Use the endpoint URL from Prometheus integration page and not from Alerts endpoint page to configure their Prometheus' Alertmanager instance
It's very easy to miss step 3 and use the "wrong" token.
If Prometheus payload is sent to the Generic JSON endpoint GitLab creates an Alert with less useful Alert details. In contrast, Prometheus payload sent to the Prometheus JSON endpoint is parsed and shown properly.
Generic JSON endpoint | Prometheus endpoint |
---|---|
Ideally, we'd like to improve user experience and make step 2 and 3 unnecessary.
Intended users
User experience goal
Don't require users to remember step 2 and 3.
Proposal
Allow generic endpoint to receive alerts from external Prometheus:
- dispatch to Prometheus specific notification service in the generic endpoint controller if the payload is Prometheus-specific
- let Prometheus specific notification service validate tokens configured via Settings > Operations > Alerts endpoint
- document this behavior in Alert Management and Prometheus integration docs
Further details
Permissions and Security
Handling Prometheus payload is guarded by the generated alert manager tokens which is not associated with a GitLab user.
Documentation
Availability & Testing
We can extend existing unit and integration specs to test the new feature.
What is the type of buyer?
GitLab Core since both Alert Management and Prometheus integration is GitLab Core.
Links / references
See for the discussion: &3360 (comment 345755346)