RousselTMLEARNING

Observability

Alerts and dashboards in ClickStack

Building charts and dashboards in ClickStack, creating alerts on a search or a chart, sending them to Slack or a webhook, and making them reliable; with equivalent SQL queries and the Grafana option.

1. Charts #

Aggregating any field.

Building a chart #

Available in: ✓ Open source ✓ Managed ClickStack

A chart starts from a source and a search, then applies an aggregation (count, sum, mean, min, max, percentiles, distinct values) on a field, optionally grouped by another (service, route, HTTP status).

ChartSource and searchAggregation
Errors per serviceLogs · level:errcount, grouped by ServiceName
p95 latency per routeTraces · service:checkout95th percentile of Duration, grouped by http.route
Active usersSessionsdistinct user IDs
HTTP codesLogs · web servicecount, grouped by http.status_code

2. Dashboards #

Gathering useful charts.

Best practices #

  • One dashboard per service or journey, built around RED signals: request rate, errors, duration.
  • Global filters (environment, service) applied to every chart on the dashboard.
  • Search tables (latest error logs) next to curves, to jump straight to details.
  • Dashboards are stored in MongoDB: back it up, or export the definition of important dashboards.

What about Grafana? #

Grafana's ClickHouse plugin queries the same tables in SQL: useful if the team already lives in Grafana or wants to mix ClickHouse with Prometheus. Both UIs can coexist on the same data.

-- Grafana panel (ClickHouse plugin): errors per service, per minute
SELECT $__timeInterval(Timestamp) AS time, ServiceName, count() AS errors
FROM otel_logs
WHERE $__timeFilter(Timestamp) AND SeverityText ILIKE 'err%'
GROUP BY time, ServiceName ORDER BY time

3. Alerts #

Being warned, not flooded.

Two types #

Alert on…Example
a saved searchmore than 50 service:checkout level:err logs in 5 minutes
a chartcheckout p95 latency above 800 ms over 10 minutes
missing datano payment service logs for 10 minutes (“below 1” threshold)

Each alert has an evaluation interval, a threshold (above or below) and destinations: Slack, e-mail, generic webhook (to Grafana IRM, PagerDuty, Opsgenie…). The message contains a link to the search or chart, hence the importance of HYPERDX_APP_URL.

Making them reliable #

  • Alert on user-visible symptoms (errors, latency) rather than on every error log.
  • Choose a window long enough to ignore a single spike (5 to 10 minutes) and a threshold based on history.
  • Add a missing-data alert on critical services: a stopped collector triggers no error alert.
  • Test each alert by triggering the condition in staging, and check the message link.
Watch out : Alerts depend on the UI (evaluation) and MongoDB (definitions): when self-hosted, monitor both, otherwise alerts silently stop.

Commands to know #

Commands to know
  • HYPERDX_APP_URL : address used in alert links
  • level:err service:checkout : search to turn into an alert
  • $__timeFilter(Timestamp) : Grafana ClickHouse plugin time filter
  • mongodump --db hyperdx : back up dashboards and alerts (self-hosted)

Sources #

Checked in November 2026: ClickStack evolves fast (Managed ClickStack is in beta), check the ClickHouse documentation before relying on a detail.

Related articles

The observability trilemma: cost, completeness and performance, and how solutions resolve it

Understanding the observability trilemma (cost, data completeness, query performance), its three trade-offs, …

Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack

Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and its variables…

Sending telemetry to ClickStack: OpenTelemetry SDKs, collectors, logs, Kubernetes and browser sessions

Feeding ClickStack: ingestion key, OpenTelemetry SDK variables, forwarding through an existing collector, fil…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!