What you will learn in this TP
- Build aggregated charts
- Assemble a filterable dashboard
- Create an alert on a saved search
- Create a missing-data alert
- Check notifications
Introduction #
What you will learn in this section
- Build aggregated charts
- Assemble a filterable dashboard
- Create an alert on a saved search
- Create a missing-data alert
- Check notifications
Context. The payment service had two incidents discovered by customers. The team wants a monitoring screen and to be warned before them, including if the service stops sending logs.
Prerequisites: a machine with Docker (4 GB free memory), curl and jq, the Lab 1 instance · Estimated time: 60 min
A four-chart dashboard, two tested alerts and their notifications received by the webhook.Setup #
Guided steps, with their solution.
1. Generate traffic and a webhook receiver
Why? An alert is tested on continuously arriving data.
Reuse the Lab 1 instance (KEYvariable), start two background generators and a webhook receiver.2. Build the charts
Why? RED signals (rate, errors, duration) summarize a service's health.
In Chart Explorer, create: spans per service (count, grouped byServiceName); p95 latency (traces, 95th percentile ofDuration); error logs per service; table of latest error logs.3. Assemble the dashboard
Why? A single screen avoids rebuilding charts at each incident.
Create thePaiement et boutiquedashboard, add the four charts, then a global filter onServiceName. Check that filtering onpaiementupdates every chart.4. Alert on errors
Why? The threshold is based on what you observe: here about 60 errors every 5 minutes.
Save theservice:paiement level:errorsearch, create an alert “above 30 in 5 minutes” to a generic webhook pointing athttp://hook:8080/alerte, then read the receiver logs.5. Alert on missing data
Why? A stopped service emits no more errors: only a missing-data alert detects it.
Create an alert “below 1 in 5 minutes” on theservice:boutiquesearch, stop thetrafic-okgenerator and wait for the notification. Then clean up.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Exploring with the ClickStack UI: sources, search, traces, sessions and SQL
Using the ClickStack UI (HyperDX): data sources, property and full-text search syntax, SQL mode, ...
Alerts and dashboards in ClickStack
Building charts and dashboards in ClickStack, creating alerts on a search or a chart, sending the...
Alerts, connectors and SLOs in Kibana
Create alert rules in Kibana (Elasticsearch query, ES|QL, custom threshold, logs), configure conn...
SigNoz, ClickStack and ClickHouse best practices for observability
Best practices for observability on ClickHouse with SigNoz or ClickStack: OpenTelemetry instrumen...
Using SigNoz: APM, traces, logs, dashboards and alerts
Use SigNoz day to day: APM view of services, traces and logs explorers, query builder, ClickHouse...
Application Observability in Grafana Cloud
Application Observability in Grafana Cloud: RED metrics from traces, service map, OpenTelemetry o...
The observability trilemma: cost, completeness and performance, and how solutions resolve it
Understanding the observability trilemma (cost, data completeness, query performance), its three ...
OpenSearch Dashboards, alerting and observability
Explore data in OpenSearch Dashboards, create alert monitors and discover Trace Analytics, Event ...
Observability on ClickHouse: why a columnar database changes the game
Why more and more observability platforms store logs, traces and metrics in ClickHouse: columnar ...