RousselTMLEARNING

TP · ClickStack: from discovery to expertise

Lab: service dashboard and alerts with correction

The correction is integrated into the TP (See the green boxes)

What you will learn in this TP
  • Build aggregated charts
  • Assemble a filterable dashboard
  • Create an alert on a saved search
  • Create a missing-data alert
  • Check notifications

Introduction #

What you will learn in this section
  • Build aggregated charts
  • Assemble a filterable dashboard
  • Create an alert on a saved search
  • Create a missing-data alert
  • Check notifications

Context. The payment service had two incidents discovered by customers. The team wants a monitoring screen and to be warned before them, including if the service stops sending logs.

Prerequisites: a machine with Docker (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 60 min

A four-chart dashboard, two tested alerts and their notifications received by the webhook.

Setup #

Guided steps, with their solution.

  1. 1. Generate traffic and a webhook receiver

    Why? An alert is tested on continuously arriving data.

    Reuse the Lab 1 instance (KEY variable), start two background generators and a webhook receiver.
    See the correction
    T='ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest'; H="authorization=\"$KEY\""
    # continuous traffic: 2 services, steady payment errors, for 15 minutes
    docker run -d --name trafic-ok  --network obs $T traces --otlp-endpoint clickstack:4317 --otlp-insecure --duration 15m --rate 10 --service boutique --otlp-header "$H"
    docker run -d --name trafic-err --network obs $T logs --otlp-endpoint clickstack:4317 --otlp-insecure --duration 15m --rate 1 \
      --service paiement --body 'timeout banque' --severity-text Error --severity-number 17 --otlp-header "$H"
    # webhook receiver: prints every request received
    docker run -d --name hook --network obs -p 8090:8080 mendhak/http-https-echo:latest
  2. 2. Build the charts

    Why? RED signals (rate, errors, duration) summarize a service's health.

    In Chart Explorer, create: spans per service (count, grouped by ServiceName); p95 latency (traces, 95th percentile of Duration); error logs per service; table of latest error logs.
    See the correction
    # Chart                 Source  Search          Aggregation             Group by
    # Throughput            Traces  (empty)         Count                   ServiceName
    # p95 latency           Traces  SpanKind:Server 95th percentile Duration ServiceName
    # Errors                Logs    level:error     Count                   ServiceName
    # Latest incidents      Logs    level:error     (search table)
    # SQL check of the displayed p95 latency:
    docker exec clickstack clickhouse-client --query "SELECT ServiceName, quantile(0.95)(Duration)/1e6 AS p95_ms FROM otel_traces WHERE Timestamp > now() - INTERVAL 15 MINUTE GROUP BY ServiceName"
  3. 3. Assemble the dashboard

    Why? A single screen avoids rebuilding charts at each incident.

    Create the Paiement et boutique dashboard, add the four charts, then a global filter on ServiceName. Check that filtering on paiement updates every chart.
    See the correction
    # Suggested layout: Throughput | p95 latency (row 1), Errors | Latest incidents (row 2)
    # Dashboard global filter:
    ServiceName:paiement
  4. 4. Alert on errors

    Why? The threshold is based on what you observe: here about 60 errors every 5 minutes.

    Save the service:paiement level:error search, create an alert “above 30 in 5 minutes” to a generic webhook pointing at http://hook:8080/alerte, then read the receiver logs.
    See the correction
    # Team Settings › Webhooks: URL http://hook:8080/alerte (the ClickStack container and “hook” are on the obs network)
    # Search › Save › Alerts: Above 30, window 5m, destination = this webhook
    docker logs --since 10m hook | grep -A20 '"path": "/alerte"'   # notification received, with the link to the search
  5. 5. Alert on missing data

    Why? A stopped service emits no more errors: only a missing-data alert detects it.

    Create an alert “below 1 in 5 minutes” on the service:boutique search, stop the trafic-ok generator and wait for the notification. Then clean up.
    See the correction
    docker stop trafic-ok
    # 5 to 6 minutes later:
    docker logs --since 10m hook | grep -c '/alerte'
    docker rm -f trafic-ok trafic-err hook

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 1 h

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Exploring with the ClickStack UI: sources, search, traces, sessions and SQL

Using the ClickStack UI (HyperDX): data sources, property and full-text search syntax, SQL mode, ...

Alerts and dashboards in ClickStack

Building charts and dashboards in ClickStack, creating alerts on a search or a chart, sending the...

Alerts, connectors and SLOs in Kibana

Create alert rules in Kibana (Elasticsearch query, ES|QL, custom threshold, logs), configure conn...

SigNoz, ClickStack and ClickHouse best practices for observability

Best practices for observability on ClickHouse with SigNoz or ClickStack: OpenTelemetry instrumen...

Using SigNoz: APM, traces, logs, dashboards and alerts

Use SigNoz day to day: APM view of services, traces and logs explorers, query builder, ClickHouse...

Application Observability in Grafana Cloud

Application Observability in Grafana Cloud: RED metrics from traces, service map, OpenTelemetry o...

The observability trilemma: cost, completeness and performance, and how solutions resolve it

Understanding the observability trilemma (cost, data completeness, query performance), its three ...

OpenSearch Dashboards, alerting and observability

Explore data in OpenSearch Dashboards, create alert monitors and discover Trace Analytics, Event ...

Observability on ClickHouse: why a columnar database changes the game

Why more and more observability platforms store logs, traces and metrics in ClickHouse: columnar ...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!