RousselTMLEARNING

Observability

Reducing, transforming and optimizing volumes with Cribl

Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop, field clean-up, Sampling, Dynamic Sampling, Suppress, Aggregations, logs to metrics, dual archive and SIEM flow, masking, and measuring reduction.

1. Where waste comes from #

What costs without serving.

Sources of useless volume #

  • Empty or redundant fields: a firewall log often has over 100 fields, many of which are never used.
  • Events with no security value: allowed traffic to internal DNS, health probes, debug logs.
  • Duplicates: the same message repeated thousands of times per minute during an incident.
  • Needless detail for the use: a dashboard showing only counts does not need every event.
Reduction example

The percentages in the diagram are illustrative: measure on your own data, gains vary widely by source.

2. The techniques #

From safest to most aggressive.

Overview #

TechniqueFunctionInformation loss
Remove empty or useless fieldsEval (Remove), Serialize (field list), Parser in “reserialize” modenone if well chosen
Drop events with no valueDrop, Regex Filtertotal for those events (archive them elsewhere)
Suppress duplicatesSuppresslow: a counter keeps track
AggregateAggregations, Rollup Metricsindividual detail
SampleSampling, Dynamic Samplingstatistical; keep for very repetitive flows
Turn into metricsPublish Metrics, Aggregationslog text

Typical settings #

# Aggregations: 1 event per minute per source/destination pair instead of thousands
time window: 60s
aggregations: [ "count().as(flows)", "sum(bytes).as(bytes)" ]
group by: [ src_ip, dst_ip, dst_port, action ]

# Suppress: one identical event every 5 minutes, with the suppressed count
key expression: `${host}:${message}`
number to allow: 1
suppression period: 300

# Dynamic Sampling: sample more what is very frequent
sample mode: logarithmic
group by key: `${status}:${uri}`

These settings are entered in function forms; the excerpt summarizes the main fields.

3. Reducing without losing anything #

The dual flow.

Full archive, reduced SIEM #

Dual flow
  1. A first non-final route sends 100% of raw data to object storage, compressed.
  2. A second route applies the reduction pipeline and sends the result to the SIEM.
  3. For an investigation, you replay the useful period from the archive to the SIEM (next chapter).
Best practice : This architecture makes reduction reversible: the security team accepts filtering far more easily when it knows it can recover everything.

4. Protecting and measuring #

Mask, then prove the gain.

Masking #

  • Mask replaces by regular expression: card numbers, e-mail addresses, tokens, passwords in URLs.
  • Hash rather than delete (C.Mask.sha256()) when you need to correlate without knowing the value.
  • Mask in the processing pipeline, before any external destination, including the archive if it leaves your perimeter.

Measuring #

  • Monitoring › Data: incoming and outgoing bytes and events per source, route, pipeline and destination.
  • Compare route input and SIEM output over a week; present the gain in GB per day and in licence cost avoided.
  • Check with the security team that detections stay identical (same alerts over the same period).
Watch out : Never drop an event type without agreement from the teams using it: a detection rule may rely on a log that looks useless.

Commands to know #

Commands to know
  • action=='allow' && dst_port==53 : typical Drop filter
  • count().as(flows) : count aggregation
  • C.Mask.sha256(email) : pseudonymize by hashing
  • Monitoring › Data › Routes : measure reduction

Sources #

Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.

Related articles

Cribl: the data pipeline for observability and security

Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl.Cloud), the…

Installing and administering Cribl Stream

Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HOME, default …

Sources, destinations and routes in Cribl Stream

Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event breaking, des…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!