The correction is integrated into the TP (See the green boxes)
What you will learn in this TP
- Deploy S3 storage and a fake SIEM
- Route a dual flow
- Drop the noise
- Deduplicate and aggregate
- Measure the reduction
Introduction #
What you will learn in this section
- Deploy S3 storage and a fake SIEM
- Route a dual flow
- Drop the noise
- Deduplicate and aggregate
- Measure the reduction
Context. The SIEM licence is exceeded every month because of firewall logs. Security accepts filtering, provided everything can be recovered and every deny stays visible.
Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance · Estimated time: 60 min
A dual flow: full compressed archive and a SIEM flow reduced by at least 50%, measured in Monitoring.Setup #
Guided steps, with their solution.
1. S3 storage and fake SIEM
Why? MinIO plays S3, an echo server plays the SIEM.
Create a shared network, MinIO with anarchivebucket and an HTTP receiversiem.See the correction
docker network create cribl-lab 2>/dev/null; docker network connect cribl-lab cribl docker run -d --name minio --network cribl-lab -p 9001:9001 \ -e MINIO_ROOT_USER=cribl -e MINIO_ROOT_PASSWORD=cribl-lab-2026 \ minio/minio:latest server /data --console-address :9001 docker run --rm --network cribl-lab --entrypoint sh minio/mc:latest -c \ 'mc alias set m http://minio:9000 cribl cribl-lab-2026 && mc mb m/archive' docker run -d --name siem --network cribl-lab mendhak/http-https-echo:latest2. Destinations
Why? Two destinations with opposite roles: keep everything, or send only what is useful.
Createout_s3_archive(Amazon S3: endpointhttp://minio:9000, bucketarchive, keyscribl/cribl-lab-2026, v4 signature, gzip JSON) andout_siem(Webhook:http://siem:8080/, JSON).See the correction
Partitioning expression (out_s3_archive): `${C.Time.strftime(_time, '%Y/%m/%d/%H')}/${sourcetype}`3. Dual-flow routes
Why? The non-final route guarantees the full archive before any reduction.
Add at the top:fw_archive(filterappname=='pan', pipelinepassthru,out_s3_archive, Final off), thenfw_siem(same filter, pipelinepl_fw_reduceto create,out_siem, Final). Send the data.See the correction
cat > gen_fw.sh <<'EOF' #!/bin/sh # 2000 lignes de pare-feu au format clé=valeur, dont beaucoup de trafic DNS autorisé et de doublons for i in $(seq 1 2000); do case $((i % 10)) in 0|1|2|3|4) l="action=allow src=10.0.$((i % 30)).$((i % 200)) dst=10.0.0.53 dport=53 proto=udp app=dns bytes=$((60 + i % 40))" ;; 5|6) l="action=allow src=10.0.$((i % 30)).$((i % 200)) dst=93.184.216.$((i % 5)) dport=443 proto=tcp app=ssl bytes=$((800 + i % 900))" ;; 7|8) l="action=deny src=203.0.113.$((i % 3)) dst=10.0.1.10 dport=22 proto=tcp app=ssh bytes=0" ;; 9) l="action=deny src=198.51.100.$((i % 50)) dst=10.0.1.20 dport=3389 proto=tcp app=rdp bytes=0" ;; esac echo "$l rule=r$((i % 4)) zone=dmz vsys=vsys1 serial=0123456789 pad1= pad2= pad3=" done EOF chmod +x gen_fw.sh ./gen_fw.sh | while read -r l; do logger -n 127.0.0.1 -P 9514 -T -t pan "$l"; done4. Parse and clean up
Why? Empty and constant fields cost without adding anything.
Inpl_fw_reduce: Parser in Extract mode, type Key=Value onmessage; then Eval removingpad*,serial,vsysand_raw.See the correction
Parser: mode Extract · type Key=Value · source field message Eval: remove fields pad*, serial, vsys, _raw, message5. Drop, Suppress, Aggregations
Why? Each function targets a different kind of waste.
Add: Drop of allowed DNS; Suppress of identical denies (key source + port, 1 per 5 minutes); Aggregations of allowed SSL traffic per minute. Denies stay visible (with a counter).See the correction
Drop filter: action=='allow' && dport==53 Suppress filter: action=='deny' key: `${src}:${dport}` allow 1 per 300 s Aggregations filter: action=='allow' && app=='ssl' window 60s aggregates: count().as(flows), sum(bytes).as(bytes) group by: src, dst, dport6. Measure
Why? The gain must be proven, and detections preserved.
Resend the 2,000 lines. Compare in Monitoring › Data › Destinations the bytes ofout_s3_archiveandout_siem, check the archive in MinIO and that every denied source/port pair reached the SIEM.See the correction
./gen_fw.sh | while read -r l; do logger -n 127.0.0.1 -P 9514 -T -t pan "$l"; done; sleep 90 docker run --rm --network cribl-lab --entrypoint sh minio/mc:latest -c \ 'mc alias set m http://minio:9000 cribl cribl-lab-2026 >/dev/null && mc du -r m/archive' docker logs siem 2>&1 | grep -o '"action":"deny"' | wc -l # expected: SIEM bytes ≤ 50 % of archive bytes; every deny src:dport present at least once
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Reducing, transforming and optimizing volumes with Cribl
Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...
Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search
Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...
Logs in Datadog: collection, pipelines, indexes and volume control
Collect logs with the agent, structure them with pipelines and processors (grok), use facets and ...
Cribl: the data pipeline for observability and security
Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...
Security with Datadog: Cloud SIEM, Cloud Security and application protection
Discover Datadog's security products: Cloud SIEM and detection rules, Cloud Security (posture, vu...
Elastic Security: SIEM, detection rules and investigation
Use Elastic Security as a SIEM: ECS data sources, detection rule types (query, EQL, ES|QL, thresh...
Reducing, masking and routing telemetry with Bindplane
Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...
Pipelines and functions: parse, enrich, mask
Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...
Cribl Edge: collecting close to the sources
Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...