Move incident closing for recovery alerts into a background job
What does this MR do and why?
Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/348676
When we receive a recovery alert notification via an alert integration, users can configure a setting to automatically close the incident associated with an alert. This currently happens synchronously before responding the alert notification request. To improve performance of the endpoint, this MR moves the logic for closing the incident into a delayed job.
How to set up and validate locally
- Have maintainer+ permissions in a project
- Enable the auto-close setting
- in the project, go to
Settings > Monitor > Alerts > Alert settings
- Enable
Automatically close associated incident when a recovery alert notification resolves an alert
- in the project, go to
- Create an alert
- go to
Settings > Monitor > Alerts
- open or create an HTTP integration
- nav to the
Send test alert
tab - Send alert with payload like
{ "title": "New alert to auto close incident" }
- go to
- Create an associated incident
- nav to
Monitor > Alerts
, open the most recent alert, and clickCreate incident
button
- nav to
- Send a recovery alert
- go to
Settings > Monitor > Alerts
- open or create an HTTP integration
- nav to the
Send test alert
tab - Send alert with payload like
{ "title": "New alert to auto close incident", "end_time": "2022-05-24T03:01:53.772Z" }
- go to
- See that the incident was automatically closed
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Sarah Yasonik