1. Charts #
Aggregating any field.
Building a chart #
Available in: ✓ Open source ✓ Managed ClickStack
A chart starts from a source and a search, then applies an aggregation (count, sum, mean, min, max, percentiles, distinct values) on a field, optionally grouped by another (service, route, HTTP status).
| Chart | Source and search | Aggregation |
|---|---|---|
| Errors per service | Logs · level:err | count, grouped by ServiceName |
| p95 latency per route | Traces · service:checkout | 95th percentile of Duration, grouped by http.route |
| Active users | Sessions | distinct user IDs |
| HTTP codes | Logs · web service | count, grouped by http.status_code |
2. Dashboards #
Gathering useful charts.
Best practices #
- One dashboard per service or journey, built around RED signals: request rate, errors, duration.
- Global filters (environment, service) applied to every chart on the dashboard.
- Search tables (latest error logs) next to curves, to jump straight to details.
- Dashboards are stored in MongoDB: back it up, or export the definition of important dashboards.
What about Grafana? #
Grafana's ClickHouse plugin queries the same tables in SQL: useful if the team already lives in Grafana or wants to mix ClickHouse with Prometheus. Both UIs can coexist on the same data.
-- Grafana panel (ClickHouse plugin): errors per service, per minute
SELECT $__timeInterval(Timestamp) AS time, ServiceName, count() AS errors
FROM otel_logs
WHERE $__timeFilter(Timestamp) AND SeverityText ILIKE 'err%'
GROUP BY time, ServiceName ORDER BY time3. Alerts #
Being warned, not flooded.
Two types #
| Alert on… | Example |
|---|---|
| a saved search | more than 50 service:checkout level:err logs in 5 minutes |
| a chart | checkout p95 latency above 800 ms over 10 minutes |
| missing data | no payment service logs for 10 minutes (“below 1” threshold) |
Each alert has an evaluation interval, a threshold (above or below) and destinations: Slack, e-mail, generic webhook (to Grafana IRM, PagerDuty, Opsgenie…). The message contains a link to the search or chart, hence the importance of HYPERDX_APP_URL.
Making them reliable #
- Alert on user-visible symptoms (errors, latency) rather than on every error log.
- Choose a window long enough to ignore a single spike (5 to 10 minutes) and a threshold based on history.
- Add a missing-data alert on critical services: a stopped collector triggers no error alert.
- Test each alert by triggering the condition in staging, and check the message link.
Watch out : Alerts depend on the UI (evaluation) and MongoDB (definitions): when self-hosted, monitor both, otherwise alerts silently stop.
Commands to know #
Commands to know
HYPERDX_APP_URL: address used in alert linkslevel:err service:checkout: search to turn into an alert$__timeFilter(Timestamp): Grafana ClickHouse plugin time filtermongodump --db hyperdx: back up dashboards and alerts (self-hosted)
Sources #
Checked in November 2026: ClickStack evolves fast (Managed ClickStack is in beta), check the ClickHouse documentation before relying on a detail.
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack
Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and i...
Telemetry in ClickHouse: schema, SQL, materialized views and retention
Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...
Sending telemetry to ClickStack: OpenTelemetry SDKs, collectors, logs, Kubernetes and browser sessions
Feeding ClickStack: ingestion key, OpenTelemetry SDK variables, forwarding through an existing co...
OpenSearch Dashboards, alerting and observability
Explore data in OpenSearch Dashboards, create alert monitors and discover Trace Analytics, Event ...
DQL in practice: exploring logs and events, notebooks, dashboards and performance
Use the Dynatrace Query Language day to day: explore logs and events, chain commands, parse field...
Kibana and Elastic Observability best practices
Checklist for running Kibana in production: versions, security, spaces and roles, versioned dashb...
Git Sync in Grafana: version dashboards in GitHub
Sync Grafana dashboards and folders with a GitHub repository: prerequisites, token, setup in Admi...
Mastering variables in Grafana: make your dashboards dynamic
Learn how to use variables to create interactive, reusable dashboards. Discover the difference be...
Application Observability in Grafana Cloud
Application Observability in Grafana Cloud: RED metrics from traces, service map, OpenTelemetry o...