RousselTMLEARNING

TP · ClickStack: from discovery to expertise

Lab: service dashboard and alerts

What you will learn in this TP
  • Build aggregated charts
  • Assemble a filterable dashboard
  • Create an alert on a saved search
  • Create a missing-data alert
  • Check notifications

Introduction #

What you will learn in this section
  • Build aggregated charts
  • Assemble a filterable dashboard
  • Create an alert on a saved search
  • Create a missing-data alert
  • Check notifications

Context. The payment service had two incidents discovered by customers. The team wants a monitoring screen and to be warned before them, including if the service stops sending logs.

Prerequisites: a machine with Docker (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 60 min

A four-chart dashboard, two tested alerts and their notifications received by the webhook.

Setup #

Guided steps, with their solution.

  1. 1. Generate traffic and a webhook receiver

    Why? An alert is tested on continuously arriving data.

    Reuse the Lab 1 instance (KEY variable), start two background generators and a webhook receiver.
  2. 2. Build the charts

    Why? RED signals (rate, errors, duration) summarize a service's health.

    In Chart Explorer, create: spans per service (count, grouped by ServiceName); p95 latency (traces, 95th percentile of Duration); error logs per service; table of latest error logs.
  3. 3. Assemble the dashboard

    Why? A single screen avoids rebuilding charts at each incident.

    Create the Paiement et boutique dashboard, add the four charts, then a global filter on ServiceName. Check that filtering on paiement updates every chart.
  4. 4. Alert on errors

    Why? The threshold is based on what you observe: here about 60 errors every 5 minutes.

    Save the service:paiement level:error search, create an alert “above 30 in 5 minutes” to a generic webhook pointing at http://hook:8080/alerte, then read the receiver logs.
  5. 5. Alert on missing data

    Why? A stopped service emits no more errors: only a missing-data alert detects it.

    Create an alert “below 1 in 5 minutes” on the service:boutique search, stop the trafic-ok generator and wait for the notification. Then clean up.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 1 h

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Operating an observability platform on ClickHouse and choosing a tool

Operate ClickHouse for telemetry (replication, disks, merges, object storage, monitoring) and com...

Datadog APM and OpenTelemetry: traces, services and correlation with logs

Instrument applications for Datadog APM (dd-trace, single step instrumentation, OpenTelemetry), u...

Telemetry in ClickHouse: schema, SQL, materialized views and retention

Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...

ClickStack: architecture, components and offerings (ClickHouse, HyperDX, OpenTelemetry)

ClickStack, ClickHouse's official observability stack: role of ClickHouse, the UI (HyperDX), the ...

Service types in Dynatrace: web, database, messaging, custom and unified services

What a service is for Dynatrace, the automatically detected service types (web requests, web serv...

Observability, SRE and incident management for DevOps teams

Operate what you ship: metrics, logs and traces, golden signals, SLIs, SLOs and error budgets, ac...

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

Datadog best practices: tags, costs, alerts and governance

Checklist for using Datadog effectively: unified tagging, cost control (hosts, logs, custom metri...

Tempo: generating metrics from traces (service graph, span metrics, exemplars)

Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!