RousselTMLEARNING

Observability

Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search

Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet formats, collectors and replay, Cribl Lake, federated search with Cribl Search and the Cribl Search destination.

1. Archiving to object storage #

The basis for replay.

A well-partitioned S3 destination #

Storing and querying
Bucket:               telemetry-archive
Key prefix:           cribl
Partitioning expr.:   `${C.Time.strftime(_time ? _time : Date.now()/1000, '%Y/%m/%d/%H')}/${sourcetype}/${host}`
Data format:          JSON   (or Parquet)
Compression:          gzip
Max file size:        32 MB   ·   Max file open time: 300 s
  • Partitioning by date then type lets you replay one hour of one log type without reading everything else.
  • Parquet (columnar) compresses better and can be queried directly by Cribl Search, Athena or ClickHouse; gzip JSON is the simplest to read back.
  • A bucket lifecycle rule (move to cold class, expiry) sets retention and cost.

2. Replaying #

Re-injecting a specific period.

Collector and replay #

Replay
  1. Create an S3 collector pointing at the archive, with a path matching the partitioning: /cribl/${_time:%Y}/${_time:%m}/${_time:%d}/${_time:%H}/${sourcetype}/${host}/.
  2. Run it on demand (Run) with the time range and a filter (host=="srv-42"): Cribl only reads the matching partitions thanks to the path variables.
  3. Replayed events go through routes like any others: add a field (replayed=true) to route them to the SIEM and tell them apart.
  4. The collector Preview mode shows a sample before running the full collection.
Best practice : Also schedule regular collections (Schedule) for APIs that do not push their data: SaaS audit, inventories, cloud metrics.

3. Cribl Lake and Cribl Search #

Storing and querying without re-ingesting.

Cribl Lake #

Cribl Lake is managed storage in Cribl.Cloud: you create datasets with their retention, and a Cribl Lake destination writes to them from Stream or Edge, without managing a bucket or format.

Cribl Search #

Cribl Search queries data where it lives: Cribl Lake, S3 buckets and some APIs, with a Kusto-like language (KQL). Since 4.17, a Cribl Search destination also lets you send data to it directly from Stream or Edge.

dataset="archive_fw" action="deny"
| summarize count() by src_ip
| sort by count_ desc
| limit 10
Good to know : The exact syntax and available datasets depend on your Cribl.Cloud workspace: start from the sample queries offered in the UI.

When to use what #

NeedSolution
Keep everything at the lowest costS3 archive + lifecycle rules
Re-inject a period into the SIEMS3 collector and replay
Find a needle without re-ingestingCribl Search
Managed storage, no bucket to administerCribl Lake

Commands to know #

Commands to know
  • C.Time.strftime(_time, '%Y/%m/%d/%H') : partition by hour
  • /${_time:%Y}/${_time:%m}/${_time:%d}/ : collector path variables
  • aws s3 ls s3://archive/cribl/2026/11/17/ --recursive | head : check the archive
  • dataset="…" | summarize count() by host : Cribl Search query

Sources #

Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.

Related articles

Cribl: the data pipeline for observability and security

Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl.Cloud), the…

Installing and administering Cribl Stream

Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HOME, default …

Sources, destinations and routes in Cribl Stream

Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event breaking, des…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!