1. From alert to incident #
A tooled journey.
Cycle #
| Step | Tool | Good practice |
|---|---|---|
| Detect | Monitors (metric, logs, APM, composite, anomalies) | Alert on the symptom (SLO, user errors), not on every cause |
| Page | On-Call (schedules, escalations) or a third-party tool (PagerDuty, Opsgenie) | One escalation policy per team, via the Software Catalog |
| Coordinate | Incident Management (roles, severity, timeline) | One incident commander, regular communication |
| Act | Workflow Automation, linked runbooks | Automate the first safe actions |
| Learn | Post-mortem generated from the incident | Follow-up actions assigned and tracked |
2. Workflow Automation #
Automating without code.
Examples #
- When a SEV-1 incident is created: create a Slack channel, invite the owning team, open a Jira ticket.
- On a « disk almost full » monitor: run a cleanup via an action or a webhook, then check the metric.
- On a security signal: isolate a user in the IdP after human validation.
Common mistake: Automatic remediation without guardrails can make an incident worse (restart loops). Provide a human approval step for destructive actions and a rate limit.
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Datadog : relier traces, logs et métriques du même service✓ solved · 1 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
Monitors, SLOs, dashboards and Datadog as code
Create effective monitors (metrics, logs, APM, composites, anomalies), SLOs, dashboards, and mana...
Datadog best practices: tags, costs, alerts and governance
Checklist for using Datadog effectively: unified tagging, cost control (hosts, logs, custom metri...
Datadog on Kubernetes: Operator, Cluster Agent and Autodiscovery
Monitor Kubernetes with Datadog: the Datadog Operator and the DatadogAgent resource, Cluster Agen...
Datadog costs and governance: usage, attribution, RBAC and tag policies
Control the bill and govern Datadog: tracking usage and estimates, cost attribution per team, usa...
Datadog dashboards and notebooks: widgets, template variables and best practices
Design useful Datadog dashboards: widget types, template variables, groups, Powerpacks, formulas,...
Advanced Datadog APM: sampling, retention, profiler and Software Catalog
Go further with Datadog APM: ingestion control and sampling, retention filters, trace metrics, Co...
RUM and Synthetic Monitoring with Datadog: synthetic transaction monitoring and user experience
Measure user experience with Datadog: browser and mobile RUM, Core Web Vitals, Session Replay, li...
Datadog: the SaaS observability platform at a glance
What Datadog is, its products (infrastructure, logs, APM, RUM, synthetics, security), its concept...
Installing the Datadog agent and configuring integrations
Install the Datadog agent on Linux, Docker and Kubernetes (Datadog Operator), understand datadog....