RousselTMLEARNING

Article · SigNoz: from discovery to expertise

Best practices

1. Collecting #

Clean data from the source.

Instrumentation #

  • OpenTelemetry only in code: the UI (SigNoz, ClickStack, Grafana) stays replaceable without re-instrumenting.
  • service.name, service.version and deployment.environment.name on every resource; stable, documented service names.
  • Structured logs (JSON) emitted with trace context: TraceId in every log is the key to correlation.
  • OpenTelemetry semantic conventions for attributes (http.route, db.system…) rather than home-made names.

Collectors #

PracticeWhy
memory_limiter then batch in every pipelineprotect the collector, insert large batches into ClickHouse
Agent per host or DaemonSet, central gatewayenrich close to the source, centralize keys and filtering
Persistent queue (file_storage)lose nothing while ClickHouse is unavailable
Tail sampling (tail_sampling)keep all errors and slow traces, sample the rest beyond a few thousand spans per second
Filter noise (filter)health probes and debug logs cost disk without value

2. Storing #

ClickHouse well tuned.

Rules #

  • Retention per signal with TTLs: for example traces 7 days, logs 14 to 30 days, aggregates 6 to 13 months.
  • Tiered storage: fast disks for recent days, object storage for history.
  • Materialized views for long-term dashboards (RED per minute), rather than rescanning raw data.
  • Replication (two replicas and ClickHouse Keeper) and tested backups (BACKUP to S3, restore in staging).
  • Do not hand-edit the tables created by the tool without following its documentation: upgrades may recreate them.

Monitoring the database #

Monitor ClickHouse independently of the platform it hosts: part count, merges, slow queries (system.query_log), free disk, replication lag. If ClickHouse goes down, the observability platform goes down with it.

3. Day-to-day use #

Searching, alerting, controlling costs.

Search and alerts #

  • Always bound the time range and filter by service: they are the sort key columns.
  • Alerts on symptoms (error rate, p95 latency, missing data), each with a link and a runbook.
  • Dashboards per service, around RED signals, with environment filters.

Security and costs #

  • Separate ClickHouse accounts: write for collectors, limited read (settings profiles) for the UI and analysts.
  • Ingestion keys treated as secrets; TLS on exposed OTLP endpoints and on the UI; single sign-on if available.
  • No personal data in attributes (e-mails, tokens): mask them in the collector (attributes or transform processors).
  • Track volume per service weekly: a chatty service is quickly spotted and fixed.

Checklist and verification #

Before go-live.

Checklist #

ItemCheck
Instrumentationevery service has service.name and its logs carry a TraceId
Collectorsotelcol_exporter_send_failed_* at zero, persistent queue configured
RetentionSHOW CREATE TABLE shows a TTL on every table
High availabilitytwo replicas or managed service; several collectors
Backupsrestore tested this quarter (ClickHouse and application database)
Alertserrors, latency and missing data on critical services, tested
Accessauthenticated UI, separate ClickHouse accounts

Commands to know #

Commands to know
  • SELECT ServiceName, count() FROM otel_logs WHERE Timestamp > now() - INTERVAL 1 DAY GROUP BY ServiceName ORDER BY 2 DESC : volume per service
  • SHOW CREATE TABLE otel_traces : check TTL and sort key
  • curl -s localhost:8888/metrics | grep send_failed : collector failed sends
  • BACKUP DATABASE … TO S3(…) : ClickHouse backup

Sources #

Checked in November 2026: ClickStack evolves fast (Managed ClickStack is in beta), check the ClickHouse documentation before relying on a detail.

📄 SigNoz, ClickStack and ClickHouse best practices for observability →

My progress

  • ⏱ Estimated time : ≈ 4 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Observability on ClickHouse: why a columnar database changes the game

Why more and more observability platforms store logs, traces and metrics in ClickHouse: columnar ...

The observability trilemma: cost, completeness and performance, and how solutions resolve it

Understanding the observability trilemma (cost, data completeness, query performance), its three ...

Operating an observability platform on ClickHouse and choosing a tool

Operate ClickHouse for telemetry (replication, disks, merges, object storage, monitoring) and com...

Using SigNoz: APM, traces, logs, dashboards and alerts

Use SigNoz day to day: APM view of services, traces and logs explorers, query builder, ClickHouse...

SigNoz: architecture and installation

SigNoz components (OpenTelemetry collector, ClickHouse, query service, interface, alerts), instal...

ClickStack: architecture, components and offerings (ClickHouse, HyperDX, OpenTelemetry)

ClickStack, ClickHouse's official observability stack: role of ClickHouse, the UI (HyperDX), the ...

Telemetry in ClickHouse: schema, SQL, materialized views and retention

Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...

Alerts and dashboards in ClickStack

Building charts and dashboards in ClickStack, creating alerts on a search or a chart, sending the...

Operating ClickHouse for ClickStack: sizing, replication, monitoring, security and backup

Running ClickHouse in production for observability: estimating volume and disk, replicating with ...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!