What you will learn in this TP
- Build aggregated charts
- Assemble a filterable dashboard
- Create an alert on a saved search
- Create a missing-data alert
- Check notifications
Introduction #
What you will learn in this section
- Build aggregated charts
- Assemble a filterable dashboard
- Create an alert on a saved search
- Create a missing-data alert
- Check notifications
Context. The payment service had two incidents discovered by customers. The team wants a monitoring screen and to be warned before them, including if the service stops sending logs.
Prerequisites: a machine with Docker (4 GB free memory), curl and jq, the Lab 1 instance · Estimated time: 60 min
A four-chart dashboard, two tested alerts and their notifications received by the webhook.Setup #
Guided steps, with their solution.
1. Generate traffic and a webhook receiver
Why? An alert is tested on continuously arriving data.
Reuse the Lab 1 instance (KEYvariable), start two background generators and a webhook receiver.2. Build the charts
Why? RED signals (rate, errors, duration) summarize a service's health.
In Chart Explorer, create: spans per service (count, grouped byServiceName); p95 latency (traces, 95th percentile ofDuration); error logs per service; table of latest error logs.3. Assemble the dashboard
Why? A single screen avoids rebuilding charts at each incident.
Create thePaiement et boutiquedashboard, add the four charts, then a global filter onServiceName. Check that filtering onpaiementupdates every chart.4. Alert on errors
Why? The threshold is based on what you observe: here about 60 errors every 5 minutes.
Save theservice:paiement level:errorsearch, create an alert “above 30 in 5 minutes” to a generic webhook pointing athttp://hook:8080/alerte, then read the receiver logs.5. Alert on missing data
Why? A stopped service emits no more errors: only a missing-data alert detects it.
Create an alert “below 1 in 5 minutes” on theservice:boutiquesearch, stop thetrafic-okgenerator and wait for the notification. Then clean up.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Operating an observability platform on ClickHouse and choosing a tool
Operate ClickHouse for telemetry (replication, disks, merges, object storage, monitoring) and com...
Datadog APM and OpenTelemetry: traces, services and correlation with logs
Instrument applications for Datadog APM (dd-trace, single step instrumentation, OpenTelemetry), u...
Telemetry in ClickHouse: schema, SQL, materialized views and retention
Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...
ClickStack: architecture, components and offerings (ClickHouse, HyperDX, OpenTelemetry)
ClickStack, ClickHouse's official observability stack: role of ClickHouse, the UI (HyperDX), the ...
Service types in Dynatrace: web, database, messaging, custom and unified services
What a service is for Dynatrace, the automatically detected service types (web requests, web serv...
Observability, SRE and incident management for DevOps teams
Operate what you ship: metrics, logs and traces, golden signals, SLIs, SLOs and error budgets, ac...
Correlating traces, metrics, logs and profiles with OpenTelemetry
Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...
Datadog best practices: tags, costs, alerts and governance
Checklist for using Datadog effectively: unified tagging, cost control (hosts, logs, custom metri...
Tempo: generating metrics from traces (service graph, span metrics, exemplars)
Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...