Monitors, SLOs and Terraform

1. Monitors #

Alerting in Datadog.

Types #

{{#is_alert}}
CPU à {{value}} % sur {{host.name}} ({{env.name}}).
Runbook : https://wiki.example.org/cpu
@slack-alertes-plateforme @pagerduty-plateforme
{{/is_alert}}
{{#is_recovery}}Retour à la normale sur {{host.name}}.{{/is_recovery}}

Good practice: Template variables ({{host.name}}) and conditional blocks allow a single « multi-alert » monitor per service instead of one per host.

2. SLOs and dashboards #

Steering.

SLOs #

A Datadog SLO is defined from monitors (uptime) or metrics (good / total ratio), with a target and a window (7, 30 or 90 days). Burn rate monitors are attached to it (see the SRE course for the concepts).

Dashboards and notebooks #

• Dashboards with template variables ($env, $service) for a single dashboard per type of service.
• Notebooks for investigations and post-mortems: graphs and text in the same place.
• Powerpacks to reuse groups of widgets.

3. Datadog as code #

Terraform.

Example #

terraform {
  required_providers { datadog = { source = "DataDog/datadog" } }
}
provider "datadog" {
  api_url = "https://api.datadoghq.eu/"      # clés : variables DD_API_KEY et DD_APP_KEY
}

resource "datadog_monitor" "cpu" {
  name    = "CPU élevé sur {{host.name}}"
  type    = "metric alert"
  query   = "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 85"
  message = "{{#is_alert}}CPU à {{value}} % sur {{host.name}}{{/is_alert}} @slack-alertes-plateforme"
  monitor_thresholds {
    critical = 85
    warning  = 75
  }
  tags = ["env:prod", "team:plateforme", "managed-by:terraform"]
}

terraform init && terraform plan && terraform apply

Good to know: Existing monitors and dashboards can be exported as JSON from the interface; tools such as terraformer generate the corresponding Terraform code to take over what already exists.