RousselTMLEARNING

Article · ClickStack: from discovery to expertise

Alerts and dashboards

1. Charts #

Aggregating any field.

Building a chart #

Available in: ✓ Open source ✓ Managed ClickStack

A chart starts from a source and a search, then applies an aggregation (count, sum, mean, min, max, percentiles, distinct values) on a field, optionally grouped by another (service, route, HTTP status).

ChartSource and searchAggregation
Errors per serviceLogs · level:errcount, grouped by ServiceName
p95 latency per routeTraces · service:checkout95th percentile of Duration, grouped by http.route
Active usersSessionsdistinct user IDs
HTTP codesLogs · web servicecount, grouped by http.status_code

2. Dashboards #

Gathering useful charts.

Best practices #

  • One dashboard per service or journey, built around RED signals: request rate, errors, duration.
  • Global filters (environment, service) applied to every chart on the dashboard.
  • Search tables (latest error logs) next to curves, to jump straight to details.
  • Dashboards are stored in MongoDB: back it up, or export the definition of important dashboards.

What about Grafana? #

Grafana's ClickHouse plugin queries the same tables in SQL: useful if the team already lives in Grafana or wants to mix ClickHouse with Prometheus. Both UIs can coexist on the same data.

-- Grafana panel (ClickHouse plugin): errors per service, per minute
SELECT $__timeInterval(Timestamp) AS time, ServiceName, count() AS errors
FROM otel_logs
WHERE $__timeFilter(Timestamp) AND SeverityText ILIKE 'err%'
GROUP BY time, ServiceName ORDER BY time

3. Alerts #

Being warned, not flooded.

Two types #

Alert on…Example
a saved searchmore than 50 service:checkout level:err logs in 5 minutes
a chartcheckout p95 latency above 800 ms over 10 minutes
missing datano payment service logs for 10 minutes (“below 1” threshold)

Each alert has an evaluation interval, a threshold (above or below) and destinations: Slack, e-mail, generic webhook (to Grafana IRM, PagerDuty, Opsgenie…). The message contains a link to the search or chart, hence the importance of HYPERDX_APP_URL.

Making them reliable #

  • Alert on user-visible symptoms (errors, latency) rather than on every error log.
  • Choose a window long enough to ignore a single spike (5 to 10 minutes) and a threshold based on history.
  • Add a missing-data alert on critical services: a stopped collector triggers no error alert.
  • Test each alert by triggering the condition in staging, and check the message link.
Watch out : Alerts depend on the UI (evaluation) and MongoDB (definitions): when self-hosted, monitor both, otherwise alerts silently stop.

Commands to know #

Commands to know
  • HYPERDX_APP_URL : address used in alert links
  • level:err service:checkout : search to turn into an alert
  • $__timeFilter(Timestamp) : Grafana ClickHouse plugin time filter
  • mongodump --db hyperdx : back up dashboards and alerts (self-hosted)

Sources #

Checked in November 2026: ClickStack evolves fast (Managed ClickStack is in beta), check the ClickHouse documentation before relying on a detail.

📄 Alerts and dashboards in ClickStack →

My progress

  • ⏱ Estimated time : ≈ 5 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack

Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and i...

Telemetry in ClickHouse: schema, SQL, materialized views and retention

Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...

Sending telemetry to ClickStack: OpenTelemetry SDKs, collectors, logs, Kubernetes and browser sessions

Feeding ClickStack: ingestion key, OpenTelemetry SDK variables, forwarding through an existing co...

OpenSearch Dashboards, alerting and observability

Explore data in OpenSearch Dashboards, create alert monitors and discover Trace Analytics, Event ...

DQL in practice: exploring logs and events, notebooks, dashboards and performance

Use the Dynatrace Query Language day to day: explore logs and events, chain commands, parse field...

Kibana and Elastic Observability best practices

Checklist for running Kibana in production: versions, security, spaces and roles, versioned dashb...

Git Sync in Grafana: version dashboards in GitHub

Sync Grafana dashboards and folders with a GitHub repository: prerequisites, token, setup in Admi...

Mastering variables in Grafana: make your dashboards dynamic

Learn how to use variables to create interactive, reusable dashboards. Discover the difference be...

Application Observability in Grafana Cloud

Application Observability in Grafana Cloud: RED metrics from traces, service map, OpenTelemetry o...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!