Delete escalation rules and notify project owners on user removal
What does this MR do?
Related issue: #335546 (closed)
This introduces logic to:
- Notify project owners when a user is removed from an escalation policy via an administrative action (user is deleted or removed from project/group)
- One email is sent to each of the project's owners for each project which has escalation rules that directly notify the deleted/removed user. Separate emails are sent for any on-call schedules the user is a part of.
- Delete direct-to-user escalation rules on user deletion/removal to avoid sending alert notifications to users without permissions
Query performance
Production records:
- Rules: ~ 200 (~ 1 is a user-based rule)
- Policies: ~ 200 (we do not expect growth to be 1:1 with rules)
- Projects/Users/Routes: Manyyyy (~ 21 mil / 10 mil / 34 mil)
Strategy:
Even with quite a bit of growth & adoption, on-call schedules/escalation rules/escalation policies won't see the scale of our more heavily used tables. User deletion/removals also take place asynchronously, so users won't ultimately see the exact impact of the queries used here. To best utilize the existing optimizations, queries added in this MR are tuned to load the needed records from the incident management tables separately from the heavily used tables.
As a result, the project/routes queries primarily utilize index scans for lookups by id. The escalation rules queries also include the escalation policy to avoid N+1s, and these generally utilize Hash Joins w/ index scans. There are also a couple sequential scans or nested loops used as well, but this is largely in scenarios where there are small number of records at play and performance is still well within reasonable.
| Production-like rules data | 50X growth | |
|---|---|---|
| Data composition |
|
|
|
incident_management_escalation_rules SELECT query
projects SELECT query
routes SELECT query
incident_management_escalation_rules DELETE query
|
incident_management_escalation_rules SELECT query
projects SELECT query
routes SELECT query
incident_management_escalation_rules DELETE query
|
|
projects SELECT query
incident_management_escalation_rules SELECT query
projects SELECT query 2
routes SELECT query
incident_management_escalation_rules DELETE query
|
projects SELECT query
incident_management_escalation_rules SELECT query
projects SELECT query 2
routes SELECT query
incident_management_escalation_rules DELETE query
|
|
incident_management_escalation_rules SELECT query
incident_management_escalation_policies SELECT query
projects SELECT query
routes SELECT query
incident_management_escalation_rules DELETE query
|
incident_management_escalation_rules SELECT query
incident_management_escalation_policies SELECT query
projects SELECT query
routes SELECT query
incident_management_escalation_rules DELETE query
|
Screenshots or screen recordings
| Modal | HTML email |
|---|---|
 |
 |
Plain text email:
Earl Marvin (colton) was removed from the following escalation policies in autodevops:
- Testing (notifying after 0 minutes unless acknowledged)
Please review the updated escalation policies for autodevops. It is recommended that you reach out to the current on-call responder to ensure continuity of on-call coverage.How to set up and validate locally
- Open & tail the logs from the gitlab directory with
tail -f log/development.log| [Skip if your gdk sends emails] - Create a user that you don't mind deleting & add them to a project (project A). (If not tailing the logs, ensure you are an owner of the project).
- Navigate to
Monitor > Escalation Policies. Add one or more escalation rules which route directly to the user you'll be deleting. Optionally, you can add whatever other rules you might want. - Repeat - add the user to another project (Project B) & add an escalation policy/rule.
- Repeat - add the user to a group (Group Z) & add an escalation policy/rule to a project (Project C) within that group.
-
Verification 1 - removing user from a group:
- Navigate to
Group information > Membersin Group Z & find the user. - Click the trash icon to remove the user from the group.
- You can see the email (both txt version & html) in the logs.
- Navigate to
Project C > Monitor > Escalation Policiesto see the policy. It should be either empty or containing any rules which did not route directly to the removed user.
- Navigate to
-
Verification 2 - removing user from a project:
- Navigate to
Project information > Membersin Project B & find the user. - Click the trash icon to remove the user from the project.
- You can see the email (both txt version & html) in the logs.
- Navigate to
Monitor > Escalation Policiesto see the policy. It should be either empty or containing any rules which did not route directly to the removed user.
- Navigate to
-
Verification 3 - deleting a user:
- Navigate to
Menu > Admin > Users& find the user. - Click the settings icon to delete the user (either deletion method should behave the same).
- You can see the email (both txt version & html) in the logs.
- Navigate to
Monitor > Escalation Policiesto see the policy. It should be either empty or containing any rules which did not route directly to the removed user.
- Navigate to
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Mayra Cabrera