What you will learn in this TP
- Deploy ClickStack with Docker Compose
- Forward file logs without loss
- Separate write and read in ClickHouse
- Tune retention and aggregates
- Back up and restore
- Validate the checklist
Introduction #
What you will learn in this section
- Deploy ClickStack with Docker Compose
- Forward file logs without loss
- Separate write and read in ClickHouse
- Tune retention and aggregates
- Back up and restore
- Validate the checklist
Context. The demo convinced: ClickStack must now receive the nginx front-end logs without losing any, with separate access, controlled retention and verified backups, before the go-live review.
Prerequisites: a machine with Docker (4 GB free memory), curl and jq, git · Estimated time: 90 min
A ClickStack Compose stack, an outage-resistant edge collector, separate accounts, per-table retention, a restorable backup and a validated checklist.Setup #
Guided steps, with their solution.
1. Deploy ClickStack with Compose
Why? One container per component: each is monitored, restarted and sized separately.
Clone the ClickStack repository, pin the image version in.env, start, create the account and export the ingestion key. Find the ClickHouse service name.2. An edge collector for nginx
Why? The persistent queue keeps logs if ClickStack is unavailable.
Writeedge.yaml, generate an nginx log file, then start a contrib collector on the Compose network.3. Test outage resistance
Why? An observability platform losing logs during an incident is useless.
Stop the ClickStack collector, add 200 lines, restart it, then check the total reaches 700 nginx lines.4. Separate accounts and limit readers
Why? An analyst must neither write nor saturate the database with an unbounded query.
Create a read-only account with a limits profile, and check it cannot drop a table.5. Retention and HTTP code aggregate
Why? Raw logs are expensive; their aggregate can be kept long for almost nothing.
Set 14 days onotel_logsand 7 days onotel_traces, then create a materialized view counting nginx requests per minute and HTTP status, kept 400 days.6. Back up ClickHouse and MongoDB
Why? A backup never restored is only a hypothesis.
Add a backup disk to ClickHouse (file mounted inconfig.dand a/backupsvolume via adocker-compose.override.yml), back upotel_logs, restore it under another name and compare. Back up MongoDB withmongodump.7. Run the checklist
Why? The go-live review asks for evidence, not impressions.
Collect evidence: collector failed sends, parts per table, slow queries, TTL, then create in the UI a missing nginx logs alert. Note what is still missing for production (replication, TLS, single sign-on).
Difficulty level : ●●●●○ (4/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack
Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and i...
Sending telemetry to ClickStack: OpenTelemetry SDKs, collectors, logs, Kubernetes and browser sessions
Feeding ClickStack: ingestion key, OpenTelemetry SDK variables, forwarding through an existing co...
Operating ClickHouse for ClickStack: sizing, replication, monitoring, security and backup
Running ClickHouse in production for observability: estimating volume and disk, replicating with ...
SigNoz, ClickStack and ClickHouse best practices for observability
Best practices for observability on ClickHouse with SigNoz or ClickStack: OpenTelemetry instrumen...
Operating an observability platform on ClickHouse and choosing a tool
Operate ClickHouse for telemetry (replication, disks, merges, object storage, monitoring) and com...
ClickStack: architecture, components and offerings (ClickHouse, HyperDX, OpenTelemetry)
ClickStack, ClickHouse's official observability stack: role of ClickHouse, the UI (HyperDX), the ...
Telemetry in ClickHouse: schema, SQL, materialized views and retention
Telemetry in ClickHouse: otel_logs, otel_traces, otel_metrics_* and hyperdx_sessions tables, sort...
Observability on ClickHouse: why a columnar database changes the game
Why more and more observability platforms store logs, traces and metrics in ClickHouse: columnar ...
Exploring with the ClickStack UI: sources, search, traces, sessions and SQL
Using the ClickStack UI (HyperDX): data sources, property and full-text search syntax, SQL mode, ...