Skip to content

Stop creating resolved alerts for self-recovering notifications

Sarah Yasonik requested to merge sy-stop-saving-self-resolving-alerts into master

What does this MR do and why?

Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/348676

When we receive a recovery alert notification via an alert integration, there's a chance that we don't find a matching, open alert in the database to resolve. For example, that might happen if the recovery alert is sent multiple times.

Before this MR, we would create and immediately resolve a new alert record for the recovery alert notification, including system notes, sending emails, etc. After this MR, we'll simply return an error response. This creates less noise for the user and minimizes database burden for us.

How to set up and validate locally

  1. Have maintainer+ permissions in a project
  2. Send a recovery notification for an alert which doesn't exist
    1. go to Settings > Monitor > Alerts
    2. open or create an HTTP integration
    3. nav to the Send test alert tab
    4. Send alert with payload like { "title": "New alert which shouldn't get created", "end_time": "2022-05-24T03:01:53.772Z" }
  3. See that an error is returned & no new alert is listed under the Resolved tab of Monitor > Alerts

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sarah Yasonik

Merge request reports

Loading