1. Where waste comes from #
What costs without serving.
Sources of useless volume #
- Empty or redundant fields: a firewall log often has over 100 fields, many of which are never used.
- Events with no security value: allowed traffic to internal DNS, health probes, debug logs.
- Duplicates: the same message repeated thousands of times per minute during an incident.
- Needless detail for the use: a dashboard showing only counts does not need every event.
The percentages in the diagram are illustrative: measure on your own data, gains vary widely by source.
2. The techniques #
From safest to most aggressive.
Overview #
| Technique | Function | Information loss |
|---|---|---|
| Remove empty or useless fields | Eval (Remove), Serialize (field list), Parser in “reserialize” mode | none if well chosen |
| Drop events with no value | Drop, Regex Filter | total for those events (archive them elsewhere) |
| Suppress duplicates | Suppress | low: a counter keeps track |
| Aggregate | Aggregations, Rollup Metrics | individual detail |
| Sample | Sampling, Dynamic Sampling | statistical; keep for very repetitive flows |
| Turn into metrics | Publish Metrics, Aggregations | log text |
Typical settings #
# Aggregations: 1 event per minute per source/destination pair instead of thousands
time window: 60s
aggregations: [ "count().as(flows)", "sum(bytes).as(bytes)" ]
group by: [ src_ip, dst_ip, dst_port, action ]
# Suppress: one identical event every 5 minutes, with the suppressed count
key expression: `${host}:${message}`
number to allow: 1
suppression period: 300
# Dynamic Sampling: sample more what is very frequent
sample mode: logarithmic
group by key: `${status}:${uri}`These settings are entered in function forms; the excerpt summarizes the main fields.
3. Reducing without losing anything #
The dual flow.
Full archive, reduced SIEM #
- A first non-final route sends 100% of raw data to object storage, compressed.
- A second route applies the reduction pipeline and sends the result to the SIEM.
- For an investigation, you replay the useful period from the archive to the SIEM (next chapter).
Best practice : This architecture makes reduction reversible: the security team accepts filtering far more easily when it knows it can recover everything.
4. Protecting and measuring #
Mask, then prove the gain.
Masking #
- Mask replaces by regular expression: card numbers, e-mail addresses, tokens, passwords in URLs.
- Hash rather than delete (
C.Mask.sha256()) when you need to correlate without knowing the value. - Mask in the processing pipeline, before any external destination, including the archive if it leaves your perimeter.
Measuring #
- Monitoring › Data: incoming and outgoing bytes and events per source, route, pipeline and destination.
- Compare route input and SIEM output over a week; present the gain in GB per day and in licence cost avoided.
- Check with the security team that detections stay identical (same alerts over the same period).
Watch out : Never drop an event type without agreement from the teams using it: a detection rule may rely on a log that looks useless.
Commands to know #
Commands to know
action=='allow' && dst_port==53: typical Drop filtercount().as(flows): count aggregationC.Mask.sha256(email): pseudonymize by hashingMonitoring › Data › Routes: measure reduction
Sources #
Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Reducing, masking and routing telemetry with Bindplane
Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...
Cribl: the data pipeline for observability and security
Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...
Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search
Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...
Cribl Stream and Edge best practices
Best practices for running Cribl in production: route and pipeline design, naming, performance, c...
Transforming with VRL and testing Vector pipelines
The VRL language (Vector Remap Language): parsing logs (JSON, regex, syslog), error handling, enr...
Cribl Edge: collecting close to the sources
Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...
Pipelines and functions: parse, enrich, mask
Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...
Sources, destinations and routes in Cribl Stream
Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...
Cribl Stream in production: distributed architecture, persistent queues and monitoring
Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...