Alerting with Alertmanager

1. Understanding Prometheus alerting #

Two components, two responsibilities.

Detect or notify? #

Prometheus evaluates PromQL expressions at regular intervals. When an expression returns series, an alert goes to the pending state, then firing after the for duration. Prometheus does not send emails itself: it forwards alerts to Alertmanager, which deduplicates, groups, routes and notifies.

2. Writing alert rules #

Rule files and good expressions.

A rule file #

# /etc/prometheus/rules/node.rules.yml
groups:
  - name: node-alerts
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.instance }} est injoignable"
          description: "Le job {{ $labels.job }} ne répond plus depuis plus de 2 minutes."

      - alert: HighCpuUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "CPU > 85 % sur {{ $labels.instance }} ({{ $value | printf \"%.0f\" }} %)"

      - alert: DiskWillFillIn4Hours
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[1h], 4*3600) < 0
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "Le disque {{ $labels.mountpoint }} de {{ $labels.instance }} sera plein dans 4 h" 

# prometheus.yml
rule_files:
  - /etc/prometheus/rules/*.rules.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

Validating and testing rules #

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/node.rules.yml
curl -X POST http://localhost:9090/-/reload       # nécessite --web.enable-lifecycle

# Tests unitaires de règles
cat > tests/node.test.yml <<'EOF'
rule_files: [../rules/node.rules.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="node", instance="web1"}'
        values: '1 1 0 0 0 0'
    alert_rule_test:
      - eval_time: 5m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels: { severity: critical, job: node, instance: web1 }
EOF
promtool test rules tests/node.test.yml

Good practice: Always use for:: without a duration, a metric that spikes once triggers a useless alert (alert fatigue).

3. Configuring Alertmanager #

Routing, grouping and receivers.

Installing with Docker Compose #

# docker-compose.yml (extrait)
  alertmanager:
    image: prom/alertmanager:latest
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command: ["--config.file=/etc/alertmanager/alertmanager.yml", "--storage.path=/alertmanager"]

Routing tree #

# alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: smtp.lab:587
  smtp_from: alertes@lab.local

route:
  receiver: equipe-ops
  group_by: [alertname, cluster]
  group_wait: 30s        # attendre avant le 1er envoi pour regrouper
  group_interval: 5m     # délai entre deux notifications d'un même groupe
  repeat_interval: 4h    # rappel si l'alerte persiste
  routes:
    - matchers: [severity="critical"]
      receiver: astreinte
      continue: true
    - matchers: [team="dba"]
      receiver: equipe-dba

receivers:
  - name: equipe-ops
    slack_configs:
      - api_url_file: /etc/alertmanager/slack_url
        channel: "#ops"
        send_resolved: true
        title: "[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}"
        text: "{{ range .Alerts }}• {{ .Annotations.summary }}\n{{ end }}"
  - name: astreinte
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd_key
  - name: equipe-dba
    email_configs:
      - to: dba@lab.local

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [alertname, instance]

Good to know: An inhibition suppresses lower-severity alerts when a more severe alert already exists for the same target: there is no point receiving « high CPU » when the host is already « unreachable ».

Validating the configuration and simulating an alert #

amtool check-config alertmanager.yml
amtool config routes show --config.file=alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=critical team=dba

# Envoyer une alerte de test sans attendre Prometheus
curl -XPOST http://localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[
  {"labels":{"alertname":"TestAlerte","severity":"warning","instance":"web1"},
   "annotations":{"summary":"Ceci est un test"}}]'

4. Day-to-day operations #

Silences, state and API.

Silences #

# Silence de 2 h pendant une maintenance
amtool silence add instance="web1" alertname="HighCpuUsage" --duration=2h --comment="mise à jour kernel" --author=martin --alertmanager.url=http://localhost:9093
amtool silence query
amtool silence expire <ID>
amtool alert query severity=critical

Alerting metrics to monitor #

# Dans Prometheus : alertes actives et état d'Alertmanager
ALERTS{alertstate="firing"}
count by (severity) (ALERTS{alertstate="firing"})
alertmanager_notifications_failed_total
rate(alertmanager_notifications_total[5m])

Common mistake: Never monitor Alertmanager only with itself: set up a « Watchdog » alert (expr: vector(1)) that must always be firing and is watched by an external service (dead man's switch).