1. Why a data pipeline? #
The problem Cribl solves.
The situation #
Companies send their logs, metrics and traces directly to their tools: SIEM, observability platform, archive. Each source is wired to each tool, each tool bills the volume it receives, and switching tools means reconfiguring hundreds of agents. A data pipeline sits between sources and destinations to route, reduce, enrich and normalize data in one place.
What you gain #
- Cost: do not index in the SIEM what is useless there (allowed traffic, empty fields, duplicates); cutting 30 to 50% of the volume sent is common.
- Freedom: change or add a tool without touching sources; send the same data to two tools during a migration.
- Quality: normalize formats, add context (team, geolocation, criticality), mask personal data before it leaves the company.
- Retention: keep 100% of raw data cheaply on object storage and replay it on demand.
2. The Cribl suite #
Five complementary products.
The products #
| Product | Role | Where it runs |
|---|---|---|
| Cribl Stream | the central pipeline: sources, routes, pipelines, destinations, replay | your servers, Kubernetes or Cribl.Cloud |
| Cribl Edge | lightweight agent: files, journald, Windows events, metrics, Kubernetes | on each host or node |
| Cribl Lake | managed dataset storage, retention per dataset | Cribl.Cloud |
| Cribl Search | federated search where data lives (S3, Lake, APIs), without re-ingesting it | Cribl.Cloud |
| Cribl.Cloud | managed control plane, workspaces, Copilot (AI assistant) | SaaS |
Good to know : This course focuses on Cribl Stream, the core of the suite, and covers Edge, Lake and Search in their collection, storage and search roles.
3. The processing model #
Sources, routes, pipelines, destinations.
An event's journey #
| Concept | Definition |
|---|---|
| Source | data entry point: listening (syslog, HEC, OTLP) or collecting (S3, REST API) |
| Route | ordered rule: a JavaScript filter, a pipeline, a destination |
| Pipeline | sequence of functions applied to events |
| Function | unit processing: Parser, Eval, Drop, Mask, Lookup, Aggregations… |
| Destination | output: Splunk, Elastic, Datadog, S3, Kafka, Loki, Cribl Lake… |
| Pack | reusable bundle of routes, pipelines and reference data |
The Cribl event #
An event is a JSON object. Some fields have special meaning:
| Field | Meaning |
|---|---|
_raw | the original event text |
_time | timestamp in seconds since the Unix epoch |
host, source, sourcetype, index | metadata inherited from the Splunk world, used to filter and route |
__inputId, __* | internal fields (double underscore): usable in filters, never sent to destinations |
cribl_* | information added by Cribl (pipeline, breaking) |
4. Licensing and versions #
Starting for free.
The offerings #
- Free: up to 1 TB per day, on Cribl.Cloud or self-hosted. Self-hosted, the free licence requires sending anonymized telemetry to Cribl and is limited to a single Worker Group.
- Paid (Standard, Enterprise): higher volumes, multiple Worker Groups, RBAC, single sign-on, support; on Cribl.Cloud, usage is counted in credits.
- Versions: a minor release about every two months (4.17 in March 2026, 4.19 in July, then 4.20); on Cribl.Cloud, a “Regular” or “Slow” release channel per workspace.
Best practice : Cribl University offers free courses and certification tracks (Cribl Certified User, then Admin Stream or Edge, then Engineer): a good complement to this course.
5. Cribl and alternatives #
Where Cribl fits.
Comparison #
| Tool | Strength | Difference with Cribl Stream |
|---|---|---|
| OpenTelemetry Collector | open standard, free, OTLP-native | YAML configuration, no UI or data preview, thinner security ecosystem |
| Bindplane | fleet management for OpenTelemetry collectors | built on OpenTelemetry; Cribl has its own engine and event format |
| Vector | very fast, open source (Datadog) | file-based configuration, no graphical control plane |
| Logstash, Fluent Bit | historical, free, widespread | no centralized multi-tool routing or built-in replay |
| Cribl Stream | UI with before/after preview, replay, 100+ integrations, SIEM-oriented | proprietary software, paid beyond 1 TB per day |
Commands to know #
Commands to know
docker run -d -p 9000:9000 cribl/cribl:latest: run Cribl Stream locally/opt/cribl/bin/cribl status: service status/opt/cribl/bin/cribl version: installed versionsourcetype=='pan:traffic': example route filter
Sources #
Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Cribl Stream and Edge best practices
Best practices for running Cribl in production: route and pipeline design, naming, performance, c...
Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search
Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...
Reducing, transforming and optimizing volumes with Cribl
Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...
Sources, destinations and routes in Cribl Stream
Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...
Cribl Edge: collecting close to the sources
Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...
Cribl Stream in production: distributed architecture, persistent queues and monitoring
Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...
Pipelines and functions: parse, enrich, mask
Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...
Installing and administering Cribl Stream
Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HO...
Automating Cribl: REST API, Packs, Git and Terraform
Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...