RousselTMLEARNING

Article · Cribl: from discovery to expertise

Reducing and optimizing volumes

1. Where waste comes from #

What costs without serving.

Sources of useless volume #

  • Empty or redundant fields: a firewall log often has over 100 fields, many of which are never used.
  • Events with no security value: allowed traffic to internal DNS, health probes, debug logs.
  • Duplicates: the same message repeated thousands of times per minute during an incident.
  • Needless detail for the use: a dashboard showing only counts does not need every event.
Reduction example

The percentages in the diagram are illustrative: measure on your own data, gains vary widely by source.

2. The techniques #

From safest to most aggressive.

Overview #

TechniqueFunctionInformation loss
Remove empty or useless fieldsEval (Remove), Serialize (field list), Parser in “reserialize” modenone if well chosen
Drop events with no valueDrop, Regex Filtertotal for those events (archive them elsewhere)
Suppress duplicatesSuppresslow: a counter keeps track
AggregateAggregations, Rollup Metricsindividual detail
SampleSampling, Dynamic Samplingstatistical; keep for very repetitive flows
Turn into metricsPublish Metrics, Aggregationslog text

Typical settings #

# Aggregations: 1 event per minute per source/destination pair instead of thousands
time window: 60s
aggregations: [ "count().as(flows)", "sum(bytes).as(bytes)" ]
group by: [ src_ip, dst_ip, dst_port, action ]

# Suppress: one identical event every 5 minutes, with the suppressed count
key expression: `${host}:${message}`
number to allow: 1
suppression period: 300

# Dynamic Sampling: sample more what is very frequent
sample mode: logarithmic
group by key: `${status}:${uri}`

These settings are entered in function forms; the excerpt summarizes the main fields.

3. Reducing without losing anything #

The dual flow.

Full archive, reduced SIEM #

Dual flow
  1. A first non-final route sends 100% of raw data to object storage, compressed.
  2. A second route applies the reduction pipeline and sends the result to the SIEM.
  3. For an investigation, you replay the useful period from the archive to the SIEM (next chapter).
Best practice : This architecture makes reduction reversible: the security team accepts filtering far more easily when it knows it can recover everything.

4. Protecting and measuring #

Mask, then prove the gain.

Masking #

  • Mask replaces by regular expression: card numbers, e-mail addresses, tokens, passwords in URLs.
  • Hash rather than delete (C.Mask.sha256()) when you need to correlate without knowing the value.
  • Mask in the processing pipeline, before any external destination, including the archive if it leaves your perimeter.

Measuring #

  • Monitoring › Data: incoming and outgoing bytes and events per source, route, pipeline and destination.
  • Compare route input and SIEM output over a week; present the gain in GB per day and in licence cost avoided.
  • Check with the security team that detections stay identical (same alerts over the same period).
Watch out : Never drop an event type without agreement from the teams using it: a detection rule may rely on a log that looks useless.

Commands to know #

Commands to know
  • action=='allow' && dst_port==53 : typical Drop filter
  • count().as(flows) : count aggregation
  • C.Mask.sha256(email) : pseudonymize by hashing
  • Monitoring › Data › Routes : measure reduction

Sources #

Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.

📄 Reducing, transforming and optimizing volumes with Cribl →

My progress

  • ⏱ Estimated time : ≈ 5 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Reducing, masking and routing telemetry with Bindplane

Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...

Cribl: the data pipeline for observability and security

Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...

Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search

Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...

Cribl Stream and Edge best practices

Best practices for running Cribl in production: route and pipeline design, naming, performance, c...

Transforming with VRL and testing Vector pipelines

The VRL language (Vector Remap Language): parsing logs (JSON, regex, syslog), error handling, enr...

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Pipelines and functions: parse, enrich, mask

Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...

Sources, destinations and routes in Cribl Stream

Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...

Cribl Stream in production: distributed architecture, persistent queues and monitoring

Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!