RousselTMLEARNING

Article · Cribl: from discovery to expertise

Archiving, replaying and searching

1. Archiving to object storage #

The basis for replay.

A well-partitioned S3 destination #

Storing and querying
Bucket:               telemetry-archive
Key prefix:           cribl
Partitioning expr.:   `${C.Time.strftime(_time ? _time : Date.now()/1000, '%Y/%m/%d/%H')}/${sourcetype}/${host}`
Data format:          JSON   (or Parquet)
Compression:          gzip
Max file size:        32 MB   ·   Max file open time: 300 s
  • Partitioning by date then type lets you replay one hour of one log type without reading everything else.
  • Parquet (columnar) compresses better and can be queried directly by Cribl Search, Athena or ClickHouse; gzip JSON is the simplest to read back.
  • A bucket lifecycle rule (move to cold class, expiry) sets retention and cost.

2. Replaying #

Re-injecting a specific period.

Collector and replay #

Replay
  1. Create an S3 collector pointing at the archive, with a path matching the partitioning: /cribl/${_time:%Y}/${_time:%m}/${_time:%d}/${_time:%H}/${sourcetype}/${host}/.
  2. Run it on demand (Run) with the time range and a filter (host=="srv-42"): Cribl only reads the matching partitions thanks to the path variables.
  3. Replayed events go through routes like any others: add a field (replayed=true) to route them to the SIEM and tell them apart.
  4. The collector Preview mode shows a sample before running the full collection.
Best practice : Also schedule regular collections (Schedule) for APIs that do not push their data: SaaS audit, inventories, cloud metrics.

3. Cribl Lake and Cribl Search #

Storing and querying without re-ingesting.

Cribl Lake #

Cribl Lake is managed storage in Cribl.Cloud: you create datasets with their retention, and a Cribl Lake destination writes to them from Stream or Edge, without managing a bucket or format.

Cribl Search #

Cribl Search queries data where it lives: Cribl Lake, S3 buckets and some APIs, with a Kusto-like language (KQL). Since 4.17, a Cribl Search destination also lets you send data to it directly from Stream or Edge.

dataset="archive_fw" action="deny"
| summarize count() by src_ip
| sort by count_ desc
| limit 10
Good to know : The exact syntax and available datasets depend on your Cribl.Cloud workspace: start from the sample queries offered in the UI.

When to use what #

NeedSolution
Keep everything at the lowest costS3 archive + lifecycle rules
Re-inject a period into the SIEMS3 collector and replay
Find a needle without re-ingestingCribl Search
Managed storage, no bucket to administerCribl Lake

Commands to know #

Commands to know
  • C.Time.strftime(_time, '%Y/%m/%d/%H') : partition by hour
  • /${_time:%Y}/${_time:%m}/${_time:%d}/ : collector path variables
  • aws s3 ls s3://archive/cribl/2026/11/17/ --recursive | head : check the archive
  • dataset="…" | summarize count() by host : Cribl Search query

Sources #

Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.

📄 Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search →

My progress

  • ⏱ Estimated time : ≈ 5 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Cribl: the data pipeline for observability and security

Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...

Reducing, transforming and optimizing volumes with Cribl

Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...

Sources, destinations and routes in Cribl Stream

Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...

Cribl Stream in production: distributed architecture, persistent queues and monitoring

Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...

Automating Cribl: REST API, Packs, Git and Terraform

Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...

Cribl Stream and Edge best practices

Best practices for running Cribl in production: route and pipeline design, naming, performance, c...

Installing and administering Cribl Stream

Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HO...

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Pipelines and functions: parse, enrich, mask

Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!