RousselTMLEARNING

TP · Cribl: from discovery to expertise

Lab: distributed cluster and persistent queue

What you will learn in this TP
  • Deploy Leader and Workers
  • Configure a group and deploy
  • Enable a persistent queue
  • Simulate a destination outage
  • Automate commit and deploy through the API

Introduction #

What you will learn in this section
  • Deploy Leader and Workers
  • Configure a group and deploy
  • Enable a persistent queue
  • Simulate a destination outage
  • Automate commit and deploy through the API

Context. Before production, the team demands two proofs: configuration deploys in a controlled way to every Worker, and a SIEM outage loses no event.

Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq  ·  Estimated time: 75 min

A three-node Cribl cluster, a version deployed through the UI then the API, and a persistent queue drained without loss after an outage.

Setup #

Guided steps, with their solution.

  1. 1. Start the cluster

    Why? The shared token authenticates Workers to the Leader.

    Write the Compose file (Leader, two Workers, fake SIEM), start it and check in Manage › Workers that both Workers are connected to the default group.
  2. 2. Configure and deploy

    Why? Nothing changes on Workers until the version is deployed.

    In the default group: Datagen source in_gen (a bundled sample, 50 events per second), Webhook destination out_siem to http://siem:8080/, route gen_siem. Click Commit then Deploy.
  3. 3. Enable the persistent queue

    Why? Without a queue, a destination outage blocks or drops data.

    On out_siem, set Backpressure behavior to Persistent Queue (max size 1 GB, gzip compression), then Commit and Deploy.
  4. 4. Simulate the outage

    Why? Only a real outage proves the behaviour.

    Stop the SIEM for 2 minutes, watch the queue grow on the Workers, restart it and check it drains.
  5. 5. Deploy through the API

    Why? In production, changes go through CI, not clicks.

    Change the in_gen rate through the API, then chain group commit, deploy and Leader commit. Check the deployed version.

Difficulty level : ●●●●○ (4/5)

My progress

  • ⏱ Estimated time : ≈ 1 h 15

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Cribl Stream in production: distributed architecture, persistent queues and monitoring

Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...

Automating Cribl: REST API, Packs, Git and Terraform

Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...

Logstash in production: persistent queues, DLQ and multiple pipelines

Make Logstash reliable in production: persistent queue, dead letter queue, multiple pipelines and...

Bindplane in production: high availability, security, Kubernetes and configuration as code

Operate Bindplane in production: active-active servers, PostgreSQL and event bus, TLS for OpAMP, ...

Logstash best practices: pipelines, performance, reliability and monitoring

Logstash best practices: multiple pipelines, dissect rather than grok, conditionals, persistent q...

Cribl Stream and Edge best practices

Best practices for running Cribl in production: route and pipeline design, naming, performance, c...

Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search

Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...

OpenTelemetry Collector: connectors, extensions and observing the Collector

Go further with the Collector: connectors (spanmetrics, servicegraph, count, routing, failover), ...

Installing and deploying ClickStack: all-in-one, Docker Compose, Helm and Managed ClickStack

Deploying ClickStack to fit the need: all-in-one container with persistence, Docker Compose and i...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!