What you will learn in this TP
- Deploy S3 storage and a fake SIEM
- Route a dual flow
- Drop the noise
- Deduplicate and aggregate
- Measure the reduction
Introduction #
What you will learn in this section
- Deploy S3 storage and a fake SIEM
- Route a dual flow
- Drop the noise
- Deduplicate and aggregate
- Measure the reduction
Context. The SIEM licence is exceeded every month because of firewall logs. Security accepts filtering, provided everything can be recovered and every deny stays visible.
Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance · Estimated time: 60 min
A dual flow: full compressed archive and a SIEM flow reduced by at least 50%, measured in Monitoring.Setup #
Guided steps, with their solution.
1. S3 storage and fake SIEM
Why? MinIO plays S3, an echo server plays the SIEM.
Create a shared network, MinIO with anarchivebucket and an HTTP receiversiem.2. Destinations
Why? Two destinations with opposite roles: keep everything, or send only what is useful.
Createout_s3_archive(Amazon S3: endpointhttp://minio:9000, bucketarchive, keyscribl/cribl-lab-2026, v4 signature, gzip JSON) andout_siem(Webhook:http://siem:8080/, JSON).3. Dual-flow routes
Why? The non-final route guarantees the full archive before any reduction.
Add at the top:fw_archive(filterappname=='pan', pipelinepassthru,out_s3_archive, Final off), thenfw_siem(same filter, pipelinepl_fw_reduceto create,out_siem, Final). Send the data.4. Parse and clean up
Why? Empty and constant fields cost without adding anything.
Inpl_fw_reduce: Parser in Extract mode, type Key=Value onmessage; then Eval removingpad*,serial,vsysand_raw.5. Drop, Suppress, Aggregations
Why? Each function targets a different kind of waste.
Add: Drop of allowed DNS; Suppress of identical denies (key source + port, 1 per 5 minutes); Aggregations of allowed SSL traffic per minute. Denies stay visible (with a counter).6. Measure
Why? The gain must be proven, and detections preserved.
Resend the 2,000 lines. Compare in Monitoring › Data › Destinations the bytes ofout_s3_archiveandout_siem, check the archive in MinIO and that every denied source/port pair reached the SIEM.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Sources, destinations and routes in Cribl Stream
Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...
Cribl Stream in production: distributed architecture, persistent queues and monitoring
Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...
Automating Cribl: REST API, Packs, Git and Terraform
Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...
Cribl Stream and Edge best practices
Best practices for running Cribl in production: route and pipeline design, naming, performance, c...
Installing and administering Cribl Stream
Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HO...
Thanos Sidecar: attaching Thanos to Prometheus and uploading to object storage
Set up Thanos Sidecar: Prometheus settings (external labels, 2h blocks), objstore configuration, ...
Advanced Datadog logs: indexes, exclusion filters, Flex Logs, archives and Sensitive Data Scanner
Control the cost and compliance of Datadog logs: pipelines and processors, indexes and retention,...
OpenTelemetry Collector: connectors, extensions and observing the Collector
Go further with the Collector: connectors (spanmetrics, servicegraph, count, routing, failover), ...
Advanced Datadog metrics: types, aggregations, cardinality and Metrics without Limits
Master Datadog metrics: types (count, rate, gauge, histogram, distribution), time and space aggre...