SigNoz, ClickStack and ClickHouse best practices for observability

1. Collecting #

OpenTelemetry first.

Principles #

• OpenTelemetry instrumentation only: the interface stays replaceable.
• service.name, service.version and deployment.environment everywhere.
• Collectors with the batch processor and a memory limit.
• Trace sampling (tail sampling) beyond a few thousand spans per second.

2. Storing #

ClickHouse operated well.

Rules #

• Per-signal retention with TTL.
• Replication and backups (BACKUP to S3).
• Tiered storage for old data.
• Monitoring of parts, merges and slow queries.
• Separate ClickHouse accounts for ingestion and reading.

Reference diagram and quick check #

Checklist.

Checklist #

1. Applications instrumented with OpenTelemetry
2. Collectors sized
3. Replicated ClickHouse with TTL
4. Alerts on critical services
5. Tested backups
6. Authenticated access to the interface