1. The chain #
Where to act.
Steps #
| Element | Role | Advice |
|---|---|---|
| Detector, metric event | Turns a metric into an event (static, adaptive, seasonal threshold) | A sliding window (3 out of 5 minutes) rather than a single point |
| Davis event | Event sent through the API or extracted from a log | Use event types for severity |
| Alerting profile | Filters notified problems: severity, duration, tags, zone | One profile per team and per on-call level |
| Integration or workflow | Sends to email, Slack, Teams, PagerDuty, ServiceNow | Workflows for complex rules |
| Maintenance window | Suspends detection or notification | Scheduled with each production release |
2. Reducing noise #
Fewer, more useful alerts.
Rules #
- Let Davis group: alert on problems, not on every event.
- Delay before notification in the profile (for example 10 minutes for low severities): many problems resolve themselves.
- Team tags set at the source to route without a manually maintained entity list.
- Monthly review of problems notified without any human action: they are candidates for removal.
Common mistake: An alerting profile without a severity filter sends minor slowdowns of every application to on-call: alert fatigue then causes the real incident to be missed.
