Skip to content

Move incident closing for recovery alerts into a background job

Sarah Yasonik requested to merge sy-add-incident-close-worker into master

What does this MR do and why?

Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/348676

When we receive a recovery alert notification via an alert integration, users can configure a setting to automatically close the incident associated with an alert. This currently happens synchronously before responding the alert notification request. To improve performance of the endpoint, this MR moves the logic for closing the incident into a delayed job.

How to set up and validate locally

  1. Have maintainer+ permissions in a project
  2. Enable the auto-close setting
    1. in the project, go to Settings > Monitor > Alerts > Alert settings
    2. Enable Automatically close associated incident when a recovery alert notification resolves an alert
  3. Create an alert
    1. go to Settings > Monitor > Alerts
    2. open or create an HTTP integration
    3. nav to the Send test alert tab
    4. Send alert with payload like { "title": "New alert to auto close incident" }
  4. Create an associated incident
    1. nav to Monitor > Alerts, open the most recent alert, and click Create incident button
  5. Send a recovery alert
    1. go to Settings > Monitor > Alerts
    2. open or create an HTTP integration
    3. nav to the Send test alert tab
    4. Send alert with payload like { "title": "New alert to auto close incident", "end_time": "2022-05-24T03:01:53.772Z" }
  6. See that the incident was automatically closed

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sarah Yasonik

Merge request reports

Loading