Skip to content

Policies and plugins to detect bugs about to breach SLO

Mark Fletcher requested to merge 426-warn-bugs-impending-slo-breaches into master

Summary

Bugs have a severity label that indicates the SLO for a fix. This automated policy aims to prompt managers about bugs in their group that are approaching the SLO threshold

  • Enable this policy to be executed daily for the gitlab-org/gitlab project
  • ~bug issues with severity1 and severity2 labels will be checked daily to see whether a comment should be posted warning the relevant PM and frontend or backend EM about the impending SLO breach

Details

This change adds a new policy and a job for the `gitlab-org/gitlab` project
to find S1/S2 bugs that are approaching the SLO for the applied severity label.

For those issues discovered the relevant EM and PM will be mentioned. Also a
label will be added to each issue affected to ensure that the rule does not repeat.
This applied label also allows bugs approaching SLO to be discovered in the issue
list.

- Detect EM and PM to mention from applied Group label
  - Pick Frontend or Backend EM based on whether `frontend` label applied
- Filter issues to comment based on applied severities
  - Currently severity::1 and severity::2
- Display breach date in comment to record in the issue when the SLO will be breached
- Apply `approaching-SLO` label to affected issues

Further Iterations

  • Expand to more projects in gitlab-org/gitlab
  • Expand to further severity labels

Action items

  • (If applicable) Add documentation to the handbook pages for Triage Operations => gitlab-com/www-gitlab-com!72606 (merged)
  • (If applicable) Identify the affected groups and how to communicate to them:
    • /cc @person_or_group =>
    • Relevant Slack channels => #product #eng-managers
    • Week-in-review

Closes #426 (closed)

Edited by Mark Fletcher

Merge request reports

Loading