1. Where waste comes from #
What costs without serving.
Sources of useless volume #
- Empty or redundant fields: a firewall log often has over 100 fields, many of which are never used.
- Events with no security value: allowed traffic to internal DNS, health probes, debug logs.
- Duplicates: the same message repeated thousands of times per minute during an incident.
- Needless detail for the use: a dashboard showing only counts does not need every event.
The percentages in the diagram are illustrative: measure on your own data, gains vary widely by source.
2. The techniques #
From safest to most aggressive.
Overview #
| Technique | Function | Information loss |
|---|---|---|
| Remove empty or useless fields | Eval (Remove), Serialize (field list), Parser in “reserialize” mode | none if well chosen |
| Drop events with no value | Drop, Regex Filter | total for those events (archive them elsewhere) |
| Suppress duplicates | Suppress | low: a counter keeps track |
| Aggregate | Aggregations, Rollup Metrics | individual detail |
| Sample | Sampling, Dynamic Sampling | statistical; keep for very repetitive flows |
| Turn into metrics | Publish Metrics, Aggregations | log text |
Typical settings #
# Aggregations: 1 event per minute per source/destination pair instead of thousands
time window: 60s
aggregations: [ "count().as(flows)", "sum(bytes).as(bytes)" ]
group by: [ src_ip, dst_ip, dst_port, action ]
# Suppress: one identical event every 5 minutes, with the suppressed count
key expression: `${host}:${message}`
number to allow: 1
suppression period: 300
# Dynamic Sampling: sample more what is very frequent
sample mode: logarithmic
group by key: `${status}:${uri}`These settings are entered in function forms; the excerpt summarizes the main fields.
3. Reducing without losing anything #
The dual flow.
Full archive, reduced SIEM #
- A first non-final route sends 100% of raw data to object storage, compressed.
- A second route applies the reduction pipeline and sends the result to the SIEM.
- For an investigation, you replay the useful period from the archive to the SIEM (next chapter).
Best practice : This architecture makes reduction reversible: the security team accepts filtering far more easily when it knows it can recover everything.
4. Protecting and measuring #
Mask, then prove the gain.
Masking #
- Mask replaces by regular expression: card numbers, e-mail addresses, tokens, passwords in URLs.
- Hash rather than delete (
C.Mask.sha256()) when you need to correlate without knowing the value. - Mask in the processing pipeline, before any external destination, including the archive if it leaves your perimeter.
Measuring #
- Monitoring › Data: incoming and outgoing bytes and events per source, route, pipeline and destination.
- Compare route input and SIEM output over a week; present the gain in GB per day and in licence cost avoided.
- Check with the security team that detections stay identical (same alerts over the same period).
Watch out : Never drop an event type without agreement from the teams using it: a detection rule may rely on a log that looks useless.
Commands to know #
Commands to know
action=='allow' && dst_port==53: typical Drop filtercount().as(flows): count aggregationC.Mask.sha256(email): pseudonymize by hashingMonitoring › Data › Routes: measure reduction
Sources #
Checked in November 2026 (Cribl 4.20): Cribl ships a minor release about every two months, check the release notes before relying on a detail.
