RousselTMLEARNING

Article · Cribl: from discovery to expertise

Cribl in production

1. Distributed architecture #

Leader, groups, Workers.

The roles #

Distributed architecture
RoleResponsibility
LeaderUI, API, Git-versioned configuration, distribution to Workers (port 4200); does not process data
Worker Groupset of Workers sharing exactly the same configuration (per region, per use)
Workernode processing data; runs several processes (usually one per core, minus one or two)

If the Leader goes down, Workers keep processing with their last configuration: only changes and centralized monitoring are interrupted.

A test cluster #

services:
  leader:
    image: cribl/cribl:4.20.0
    environment:
      CRIBL_DIST_MODE: leader
      CRIBL_DIST_LEADER_URL: tcp://0.0.0.0:4200
      CRIBL_DIST_AUTH_TOKENS: ${CRIBL_TOKEN}
      CRIBL_VOLUME_DIR: /opt/cribl/config-volume
    ports: ["19000:9000"]
    volumes: ["./leader:/opt/cribl/config-volume"]
  workers:
    image: cribl/cribl:4.20.0
    environment:
      CRIBL_DIST_MODE: worker
      CRIBL_DIST_LEADER_URL: tcp://${CRIBL_TOKEN}@leader:4200
    depends_on: [leader]
# docker compose up -d --scale workers=2
Good to know : With the free licence, only one group (default) is available. The CRIBL_DIST_AUTH_TOKENS token authenticates Workers: choose a random value and keep it secret.

2. Changing safely #

Commit and deploy.

The cycle #

  1. You change a group's configuration on the Leader: nothing changes on Workers yet.
  2. Commit: the change becomes a Git version, with a message.
  3. Deploy: the group's Workers download the version and reload without cutting the flow.
  4. If something goes wrong, you redeploy a previous version from the history.
Best practice : Configure a remote Git repository (Settings › Git) to push history automatically: it is your configuration backup and audit trail.

3. Scaling #

Processes, Workers, balancing.

Principles #

  • Vertically: more cores = more Worker processes; configurable in group settings.
  • Horizontally: add Workers to the group; for push sources (syslog, HEC), put a load balancer in front of them.
  • Pull sources (Kafka, SQS) spread naturally across Workers; collectors distribute their tasks.
  • Size from measurements: CPU per process, throughput per Worker, and keep headroom for spikes and the loss of a node.
Watch out : UDP syslog behind a load balancer loses flow affinity: prefer TCP, or a local syslog relay forwarding over TCP.

4. Losing nothing #

Backpressure and persistent queues.

Persistent queues #

Persistent queue
Destination settingEffect when the destination cannot keep up
BlockCribl slows source reading (senders wait); risk of loss on the UDP sender side
Dropevents are discarded: acceptable for low-criticality data
Persistent Queueevents are written to disk, then sent in order when the destination returns
  • Size the maximum queue size (GB) to cover the longest tolerated outage, and monitor disk space.
  • Push sources can also have a persistent queue, useful when the sender cannot retry; since 4.20, the “Smart” mode of source queues is deprecated in favour of always-on mode.

Leader high availability #

A standby second Leader, with shared storage (NFS) for configuration, takes over on failure. On Cribl.Cloud, the Leader is managed by Cribl.

5. Monitoring and securing #

Seeing and protecting.

Monitoring #

  • Monitoring: throughput per source and destination, Worker CPU and memory, persistent queues, errors.
  • Cribl Internal source: send Cribl's internal logs and metrics to your observability tool, to alert even if the UI is unavailable.
  • Notifications: alerts on silent sources, failing destinations, filling queues, licence.
  • Health endpoint: GET /api/v1/health for load balancer and Kubernetes probes.

Securing #

  • TLS on the UI, between Leader and Workers, and on exposed sources.
  • Named accounts, roles (RBAC depending on licence), single sign-on; no shared admin account.
  • Secrets (destination passwords, keys) in Cribl's secrets manager, not in plain text in pipelines.

Commands to know #

Commands to know
  • docker compose up -d --scale workers=2 : add Workers (test)
  • curl -s http://leader:9000/api/v1/health : node health
  • du -sh /opt/cribl/state/queues : persistent queue size
  • CRIBL_DIST_LEADER_URL=tcp://<jeton>@leader:4200 : attach a Worker

Sources #

Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.

📄 Cribl Stream in production: distributed architecture, persistent queues and monitoring →

My progress

  • ⏱ Estimated time : ≈ 7 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Pipelines and functions: parse, enrich, mask

Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...

Logstash best practices: pipelines, performance, reliability and monitoring

Logstash best practices: multiple pipelines, dissect rather than grok, conditionals, persistent q...

Bindplane best practices

Checklist for operating Bindplane: organizing by labels and fleets, measured reduction, masking a...

Arize in production: self-hosted Phoenix, Arize AX, security, costs and monitoring

Going to production: Phoenix with PostgreSQL and authentication, an OpenTelemetry Collector in fr...

VictoriaMetrics best practices

Production checklist for VictoriaMetrics: sizing, retention, high availability, vmagent, cardinal...

HAProxy in Kubernetes: the high-performance Ingress Controller

Discover in depth how HAProxy integrates with Kubernetes: direct-to-pod architecture, hot update ...

Grafana Mimir best practices: architecture, limits, costs and operations

Production checklist for Grafana Mimir: architecture, zone-aware replication, per-tenant limits, ...

Alertmanager high availability and operations

Set up a highly available Alertmanager cluster (gossip), configure Prometheus for all members, mo...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!