RousselTMLEARNING

TP · ClickStack: from discovery to expertise

Lab: service dashboard and alerts

What you will learn in this TP
  • Build aggregated charts
  • Assemble a filterable dashboard
  • Create an alert on a saved search
  • Create a missing-data alert
  • Check notifications

Introduction #

What you will learn in this section
  • Build aggregated charts
  • Assemble a filterable dashboard
  • Create an alert on a saved search
  • Create a missing-data alert
  • Check notifications

Context. The payment service had two incidents discovered by customers. The team wants a monitoring screen and to be warned before them, including if the service stops sending logs.

Prerequisites: a machine with Docker (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 60 min

A four-chart dashboard, two tested alerts and their notifications received by the webhook.

Setup #

Guided steps, with their solution.

  1. 1. Generate traffic and a webhook receiver

    Why? An alert is tested on continuously arriving data.

    Reuse the Lab 1 instance (KEY variable), start two background generators and a webhook receiver.
  2. 2. Build the charts

    Why? RED signals (rate, errors, duration) summarize a service's health.

    In Chart Explorer, create: spans per service (count, grouped by ServiceName); p95 latency (traces, 95th percentile of Duration); error logs per service; table of latest error logs.
  3. 3. Assemble the dashboard

    Why? A single screen avoids rebuilding charts at each incident.

    Create the Paiement et boutique dashboard, add the four charts, then a global filter on ServiceName. Check that filtering on paiement updates every chart.
  4. 4. Alert on errors

    Why? The threshold is based on what you observe: here about 60 errors every 5 minutes.

    Save the service:paiement level:error search, create an alert “above 30 in 5 minutes” to a generic webhook pointing at http://hook:8080/alerte, then read the receiver logs.
  5. 5. Alert on missing data

    Why? A stopped service emits no more errors: only a missing-data alert detects it.

    Create an alert “below 1 in 5 minutes” on the service:boutique search, stop the trafic-ok generator and wait for the notification. Then clean up.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 1 h

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Exploring with the ClickStack UI: sources, search, traces, sessions and SQL

Using the ClickStack UI (HyperDX): data sources, property and full-text search syntax, SQL mode, ...

Alerts and dashboards in ClickStack

Building charts and dashboards in ClickStack, creating alerts on a search or a chart, sending the...

Alerts, connectors and SLOs in Kibana

Create alert rules in Kibana (Elasticsearch query, ES|QL, custom threshold, logs), configure conn...

SigNoz, ClickStack and ClickHouse best practices for observability

Best practices for observability on ClickHouse with SigNoz or ClickStack: OpenTelemetry instrumen...

Using SigNoz: APM, traces, logs, dashboards and alerts

Use SigNoz day to day: APM view of services, traces and logs explorers, query builder, ClickHouse...

Application Observability in Grafana Cloud

Application Observability in Grafana Cloud: RED metrics from traces, service map, OpenTelemetry o...

The observability trilemma: cost, completeness and performance, and how solutions resolve it

Understanding the observability trilemma (cost, data completeness, query performance), its three ...

OpenSearch Dashboards, alerting and observability

Explore data in OpenSearch Dashboards, create alert monitors and discover Trace Analytics, Event ...

Observability on ClickHouse: why a columnar database changes the game

Why more and more observability platforms store logs, traces and metrics in ClickHouse: columnar ...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!