The correction is integrated into the TP (See the green boxes)
What you will learn in this TP
- Deploy Leader and Workers
- Configure a group and deploy
- Enable a persistent queue
- Simulate a destination outage
- Automate commit and deploy through the API
Introduction #
What you will learn in this section
- Deploy Leader and Workers
- Configure a group and deploy
- Enable a persistent queue
- Simulate a destination outage
- Automate commit and deploy through the API
Context. Before production, the team demands two proofs: configuration deploys in a controlled way to every Worker, and a SIEM outage loses no event.
Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq · Estimated time: 75 min
A three-node Cribl cluster, a version deployed through the UI then the API, and a persistent queue drained without loss after an outage.Setup #
Guided steps, with their solution.
1. Start the cluster
Why? The shared token authenticates Workers to the Leader.
Write the Compose file (Leader, two Workers, fake SIEM), start it and check in Manage › Workers that both Workers are connected to thedefaultgroup.See the correction
mkdir cribl-dist && cd cribl-dist export CRIBL_TOKEN=$(openssl rand -hex 16); echo "CRIBL_TOKEN=$CRIBL_TOKEN" > .env cat > docker-compose.yml <<'EOF' services: leader: image: cribl/cribl:latest environment: CRIBL_DIST_MODE: leader CRIBL_DIST_LEADER_URL: tcp://0.0.0.0:4200 CRIBL_DIST_AUTH_TOKENS: ${CRIBL_TOKEN} CRIBL_VOLUME_DIR: /opt/cribl/config-volume ports: ["19000:9000"] volumes: ["./leader:/opt/cribl/config-volume"] workers: image: cribl/cribl:latest environment: CRIBL_DIST_MODE: worker CRIBL_DIST_LEADER_URL: tcp://${CRIBL_TOKEN}@leader:4200 depends_on: [leader] siem: image: mendhak/http-https-echo:latest EOF docker compose up -d --scale workers=2 until curl -sf localhost:19000/api/v1/health >/dev/null; do sleep 3; done; docker compose ps2. Configure and deploy
Why? Nothing changes on Workers until the version is deployed.
In thedefaultgroup: Datagen sourcein_gen(a bundled sample, 50 events per second), Webhook destinationout_siemtohttp://siem:8080/, routegen_siem. Click Commit then Deploy.See the correction
docker compose logs --since 1m siem | grep -c '"path"' # > 0 : both Workers send to the SIEM (see Monitoring › Data › Destinations)3. Enable the persistent queue
Why? Without a queue, a destination outage blocks or drops data.
Onout_siem, set Backpressure behavior to Persistent Queue (max size 1 GB, gzip compression), then Commit and Deploy.See the correction
Backpressure behavior: Persistent Queue Max queue size: 1GB · Compression: gzip · Queue path: $CRIBL_HOME/state/queues4. Simulate the outage
Why? Only a real outage proves the behaviour.
Stop the SIEM for 2 minutes, watch the queue grow on the Workers, restart it and check it drains.See the correction
docker compose stop siem; sleep 120 for w in $(docker compose ps -q workers); do docker exec $w du -sh /opt/cribl/state/queues; done docker compose start siem; sleep 60 for w in $(docker compose ps -q workers); do docker exec $w du -sh /opt/cribl/state/queues; done5. Deploy through the API
Why? In production, changes go through CI, not clicks.
Change thein_genrate through the API, then chain group commit, deploy and Leader commit. Check the deployed version.See the correction
CRIBL=http://localhost:19000 TOKEN=$(curl -s -X POST $CRIBL/api/v1/auth/login -H 'Content-Type: application/json' -d '{"username":"admin","password":"'"$CRIBL_PASSWORD"'"}' | jq -r .token) H=(-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json') IN=$(curl -s "${H[@]}" $CRIBL/api/v1/m/default/system/inputs/in_gen | jq '.items[0] | .samples[0].eventsPerSec = 20') curl -s "${H[@]}" -X PATCH $CRIBL/api/v1/m/default/system/inputs/in_gen -d "$IN" | jq -r '.items[0].id' COMMIT=$(curl -s "${H[@]}" -X POST $CRIBL/api/v1/version/commit -d '{"message":"api: in_gen 20 eps","group":"default"}' | jq -r '.items[0].commit') curl -s "${H[@]}" -X PATCH $CRIBL/api/v1/products/stream/groups/default/deploy -d "{\"version\":\"$COMMIT\"}" | jq -r '.items[0].configVersion' curl -s "${H[@]}" -X POST $CRIBL/api/v1/version/commit -d '{"message":"sync leader"}' >/dev/null
Difficulty level : ●●●●○ (4/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Cribl Stream in production: distributed architecture, persistent queues and monitoring
Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...
Automating Cribl: REST API, Packs, Git and Terraform
Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...
Logstash in production: persistent queues, DLQ and multiple pipelines
Make Logstash reliable in production: persistent queue, dead letter queue, multiple pipelines and...
Bindplane in production: high availability, security, Kubernetes and configuration as code
Operate Bindplane in production: active-active servers, PostgreSQL and event bus, TLS for OpAMP, ...
Logstash best practices: pipelines, performance, reliability and monitoring
Logstash best practices: multiple pipelines, dissect rather than grok, conditionals, persistent q...
Cribl Stream and Edge best practices
Best practices for running Cribl in production: route and pipeline design, naming, performance, c...
Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search
Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...
OpenTelemetry Collector: connectors, extensions and observing the Collector
Go further with the Collector: connectors (spanmetrics, servicegraph, count, routing, failover), ...
Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack
Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and i...