1. Why a data pipeline? #
The problem Cribl solves.
The situation #
Companies send their logs, metrics and traces directly to their tools: SIEM, observability platform, archive. Each source is wired to each tool, each tool bills the volume it receives, and switching tools means reconfiguring hundreds of agents. A data pipeline sits between sources and destinations to route, reduce, enrich and normalize data in one place.
What you gain #
- Cost: do not index in the SIEM what is useless there (allowed traffic, empty fields, duplicates); cutting 30 to 50% of the volume sent is common.
- Freedom: change or add a tool without touching sources; send the same data to two tools during a migration.
- Quality: normalize formats, add context (team, geolocation, criticality), mask personal data before it leaves the company.
- Retention: keep 100% of raw data cheaply on object storage and replay it on demand.
2. The Cribl suite #
Five complementary products.
The products #
| Product | Role | Where it runs |
|---|---|---|
| Cribl Stream | the central pipeline: sources, routes, pipelines, destinations, replay | your servers, Kubernetes or Cribl.Cloud |
| Cribl Edge | lightweight agent: files, journald, Windows events, metrics, Kubernetes | on each host or node |
| Cribl Lake | managed dataset storage, retention per dataset | Cribl.Cloud |
| Cribl Search | federated search where data lives (S3, Lake, APIs), without re-ingesting it | Cribl.Cloud |
| Cribl.Cloud | managed control plane, workspaces, Copilot (AI assistant) | SaaS |
Good to know : This course focuses on Cribl Stream, the core of the suite, and covers Edge, Lake and Search in their collection, storage and search roles.
3. The processing model #
Sources, routes, pipelines, destinations.
An event's journey #
| Concept | Definition |
|---|---|
| Source | data entry point: listening (syslog, HEC, OTLP) or collecting (S3, REST API) |
| Route | ordered rule: a JavaScript filter, a pipeline, a destination |
| Pipeline | sequence of functions applied to events |
| Function | unit processing: Parser, Eval, Drop, Mask, Lookup, Aggregations… |
| Destination | output: Splunk, Elastic, Datadog, S3, Kafka, Loki, Cribl Lake… |
| Pack | reusable bundle of routes, pipelines and reference data |
The Cribl event #
An event is a JSON object. Some fields have special meaning:
| Field | Meaning |
|---|---|
_raw | the original event text |
_time | timestamp in seconds since the Unix epoch |
host, source, sourcetype, index | metadata inherited from the Splunk world, used to filter and route |
__inputId, __* | internal fields (double underscore): usable in filters, never sent to destinations |
cribl_* | information added by Cribl (pipeline, breaking) |
4. Licensing and versions #
Starting for free.
The offerings #
- Free: up to 1 TB per day, on Cribl.Cloud or self-hosted. Self-hosted, the free licence requires sending anonymized telemetry to Cribl and is limited to a single Worker Group.
- Paid (Standard, Enterprise): higher volumes, multiple Worker Groups, RBAC, single sign-on, support; on Cribl.Cloud, usage is counted in credits.
- Versions: a minor release about every two months (4.17 in March 2026, 4.19 in July, then 4.20); on Cribl.Cloud, a “Regular” or “Slow” release channel per workspace.
Best practice : Cribl University offers free courses and certification tracks (Cribl Certified User, then Admin Stream or Edge, then Engineer): a good complement to this course.
5. Cribl and alternatives #
Where Cribl fits.
Comparison #
| Tool | Strength | Difference with Cribl Stream |
|---|---|---|
| OpenTelemetry Collector | open standard, free, OTLP-native | YAML configuration, no UI or data preview, thinner security ecosystem |
| Bindplane | fleet management for OpenTelemetry collectors | built on OpenTelemetry; Cribl has its own engine and event format |
| Vector | very fast, open source (Datadog) | file-based configuration, no graphical control plane |
| Logstash, Fluent Bit | historical, free, widespread | no centralized multi-tool routing or built-in replay |
| Cribl Stream | UI with before/after preview, replay, 100+ integrations, SIEM-oriented | proprietary software, paid beyond 1 TB per day |
Commands to know #
Commands to know
docker run -d -p 9000:9000 cribl/cribl:latest: run Cribl Stream locally/opt/cribl/bin/cribl status: service status/opt/cribl/bin/cribl version: installed versionsourcetype=='pan:traffic': example route filter
Sources #
Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Bindplane best practices
Checklist for operating Bindplane: organizing by labels and fleets, measured reduction, masking a...
Grafana Alloy: Log Management with Loki
Discover how to configure Grafana Alloy to read log files, journald, or network streams, process ...
Security with Datadog: Cloud SIEM, Cloud Security and application protection
Discover Datadog's security products: Cloud SIEM and detection rules, Cloud Security (posture, vu...
Agent policies, integrations, data streams and pipelines
Build agent policies, add integrations (System, Nginx, Custom Logs), choose namespaces, customise...
Vector: transforms in detail (remap and VRL, route, filter, reduction and metrics)
Transform events in Vector: remap and the VRL language (parsing, enrichment, error handling), rou...
OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines
Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...
Dynatrace security and governance: advanced IAM, policies, SSO, audit and data protection
Secure and govern Dynatrace: SSO and SCIM, groups, IAM policies with conditions, security context...
Reducing, masking and routing telemetry with Bindplane
Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...
Advanced Datadog logs: indexes, exclusion filters, Flex Logs, archives and Sensitive Data Scanner
Control the cost and compliance of Datadog logs: pipelines and processors, indexes and retention,...