RousselTMLEARNING

TP · Cribl: from discovery to expertise

Lab: halving the volume sent to the SIEM

What you will learn in this TP
  • Deploy S3 storage and a fake SIEM
  • Route a dual flow
  • Drop the noise
  • Deduplicate and aggregate
  • Measure the reduction

Introduction #

What you will learn in this section
  • Deploy S3 storage and a fake SIEM
  • Route a dual flow
  • Drop the noise
  • Deduplicate and aggregate
  • Measure the reduction

Context. The SIEM licence is exceeded every month because of firewall logs. Security accepts filtering, provided everything can be recovered and every deny stays visible.

Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 60 min

A dual flow: full compressed archive and a SIEM flow reduced by at least 50%, measured in Monitoring.

Setup #

Guided steps, with their solution.

  1. 1. S3 storage and fake SIEM

    Why? MinIO plays S3, an echo server plays the SIEM.

    Create a shared network, MinIO with an archive bucket and an HTTP receiver siem.
  2. 2. Destinations

    Why? Two destinations with opposite roles: keep everything, or send only what is useful.

    Create out_s3_archive (Amazon S3: endpoint http://minio:9000, bucket archive, keys cribl / cribl-lab-2026, v4 signature, gzip JSON) and out_siem (Webhook: http://siem:8080/, JSON).
  3. 3. Dual-flow routes

    Why? The non-final route guarantees the full archive before any reduction.

    Add at the top: fw_archive (filter appname=='pan', pipeline passthru, out_s3_archive, Final off), then fw_siem (same filter, pipeline pl_fw_reduce to create, out_siem, Final). Send the data.
  4. 4. Parse and clean up

    Why? Empty and constant fields cost without adding anything.

    In pl_fw_reduce: Parser in Extract mode, type Key=Value on message; then Eval removing pad*, serial, vsys and _raw.
  5. 5. Drop, Suppress, Aggregations

    Why? Each function targets a different kind of waste.

    Add: Drop of allowed DNS; Suppress of identical denies (key source + port, 1 per 5 minutes); Aggregations of allowed SSL traffic per minute. Denies stay visible (with a counter).
  6. 6. Measure

    Why? The gain must be proven, and detections preserved.

    Resend the 2,000 lines. Compare in Monitoring › Data › Destinations the bytes of out_s3_archive and out_siem, check the archive in MinIO and that every denied source/port pair reached the SIEM.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 1 h

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Sources, destinations and routes in Cribl Stream

Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...

Cribl Stream in production: distributed architecture, persistent queues and monitoring

Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...

Automating Cribl: REST API, Packs, Git and Terraform

Managing Cribl as code: REST API (authentication, group paths, commit and deploy), reusable Packs...

Cribl Stream and Edge best practices

Best practices for running Cribl in production: route and pipeline design, naming, performance, c...

Installing and administering Cribl Stream

Installing Cribl Stream with Docker or on Linux, running it as a service, understanding $CRIBL_HO...

Thanos Sidecar: attaching Thanos to Prometheus and uploading to object storage

Set up Thanos Sidecar: Prometheus settings (external labels, 2h blocks), objstore configuration, ...

Advanced Datadog logs: indexes, exclusion filters, Flex Logs, archives and Sensitive Data Scanner

Control the cost and compliance of Datadog logs: pipelines and processors, indexes and retention,...

OpenTelemetry Collector: connectors, extensions and observing the Collector

Go further with the Collector: connectors (spanmetrics, servicegraph, count, routing, failover), ...

Advanced Datadog metrics: types, aggregations, cardinality and Metrics without Limits

Master Datadog metrics: types (count, rate, gauge, histogram, distribution), time and space aggre...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!