1. Collecting #
Clean data from the source.
Instrumentation #
- OpenTelemetry only in code: the UI (SigNoz, ClickStack, Grafana) stays replaceable without re-instrumenting.
service.name,service.versionanddeployment.environment.nameon every resource; stable, documented service names.- Structured logs (JSON) emitted with trace context:
TraceIdin every log is the key to correlation. - OpenTelemetry semantic conventions for attributes (
http.route,db.system…) rather than home-made names.
Collectors #
| Practice | Why |
|---|---|
memory_limiter then batch in every pipeline | protect the collector, insert large batches into ClickHouse |
| Agent per host or DaemonSet, central gateway | enrich close to the source, centralize keys and filtering |
Persistent queue (file_storage) | lose nothing while ClickHouse is unavailable |
Tail sampling (tail_sampling) | keep all errors and slow traces, sample the rest beyond a few thousand spans per second |
Filter noise (filter) | health probes and debug logs cost disk without value |
2. Storing #
ClickHouse well tuned.
Rules #
- Retention per signal with TTLs: for example traces 7 days, logs 14 to 30 days, aggregates 6 to 13 months.
- Tiered storage: fast disks for recent days, object storage for history.
- Materialized views for long-term dashboards (RED per minute), rather than rescanning raw data.
- Replication (two replicas and ClickHouse Keeper) and tested backups (
BACKUPto S3, restore in staging). - Do not hand-edit the tables created by the tool without following its documentation: upgrades may recreate them.
Monitoring the database #
Monitor ClickHouse independently of the platform it hosts: part count, merges, slow queries (system.query_log), free disk, replication lag. If ClickHouse goes down, the observability platform goes down with it.
3. Day-to-day use #
Searching, alerting, controlling costs.
Search and alerts #
- Always bound the time range and filter by service: they are the sort key columns.
- Alerts on symptoms (error rate, p95 latency, missing data), each with a link and a runbook.
- Dashboards per service, around RED signals, with environment filters.
Security and costs #
- Separate ClickHouse accounts: write for collectors, limited read (settings profiles) for the UI and analysts.
- Ingestion keys treated as secrets; TLS on exposed OTLP endpoints and on the UI; single sign-on if available.
- No personal data in attributes (e-mails, tokens): mask them in the collector (
attributesortransformprocessors). - Track volume per service weekly: a chatty service is quickly spotted and fixed.
Checklist and verification #
Before go-live.
Checklist #
| Item | Check |
|---|---|
| Instrumentation | every service has service.name and its logs carry a TraceId |
| Collectors | otelcol_exporter_send_failed_* at zero, persistent queue configured |
| Retention | SHOW CREATE TABLE shows a TTL on every table |
| High availability | two replicas or managed service; several collectors |
| Backups | restore tested this quarter (ClickHouse and application database) |
| Alerts | errors, latency and missing data on critical services, tested |
| Access | authenticated UI, separate ClickHouse accounts |
Commands to know #
Commands to know
SELECT ServiceName, count() FROM otel_logs WHERE Timestamp > now() - INTERVAL 1 DAY GROUP BY ServiceName ORDER BY 2 DESC: volume per serviceSHOW CREATE TABLE otel_traces: check TTL and sort keycurl -s localhost:8888/metrics | grep send_failed: collector failed sendsBACKUP DATABASE … TO S3(…): ClickHouse backup
Sources #
Checked in November 2026: ClickStack evolves fast (Managed ClickStack is in beta), check the ClickHouse documentation before relying on a detail.
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Observability on ClickHouse: why a columnar database changes the game
Why more and more observability platforms store logs, traces and metrics in ClickHouse: columnar ...
Operating an observability platform on ClickHouse and choosing a tool
Operate ClickHouse for telemetry (replication, disks, merges, object storage, monitoring) and com...
Operating ClickHouse for ClickStack: sizing, replication, monitoring, security and backup
Running ClickHouse in production for observability: estimating volume and disk, replicating with ...
The observability trilemma: cost, completeness and performance, and how solutions resolve it
Understanding the observability trilemma (cost, data completeness, query performance), its three ...
ClickStack: architecture, components and offerings (ClickHouse, HyperDX, OpenTelemetry)
ClickStack, ClickHouse's official observability stack: role of ClickHouse, the UI (HyperDX), the ...
Alerts and dashboards in ClickStack
Building charts and dashboards in ClickStack, creating alerts on a search or a chart, sending the...
Telemetry in ClickHouse: schema, SQL, materialized views and retention
Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...
Sending telemetry to ClickStack: OpenTelemetry SDKs, collectors, logs, Kubernetes and browser sessions
Feeding ClickStack: ingestion key, OpenTelemetry SDK variables, forwarding through an existing co...
Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack
Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and i...