RousselTMLEARNING

TP · Cribl: from discovery to expertise

Lab: halving the volume sent to the SIEM

What you will learn in this TP
  • Deploy S3 storage and a fake SIEM
  • Route a dual flow
  • Drop the noise
  • Deduplicate and aggregate
  • Measure the reduction

Introduction #

What you will learn in this section
  • Deploy S3 storage and a fake SIEM
  • Route a dual flow
  • Drop the noise
  • Deduplicate and aggregate
  • Measure the reduction

Context. The SIEM licence is exceeded every month because of firewall logs. Security accepts filtering, provided everything can be recovered and every deny stays visible.

Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 60 min

A dual flow: full compressed archive and a SIEM flow reduced by at least 50%, measured in Monitoring.

Setup #

Guided steps, with their solution.

  1. 1. S3 storage and fake SIEM

    Why? MinIO plays S3, an echo server plays the SIEM.

    Create a shared network, MinIO with an archive bucket and an HTTP receiver siem.
  2. 2. Destinations

    Why? Two destinations with opposite roles: keep everything, or send only what is useful.

    Create out_s3_archive (Amazon S3: endpoint http://minio:9000, bucket archive, keys cribl / cribl-lab-2026, v4 signature, gzip JSON) and out_siem (Webhook: http://siem:8080/, JSON).
  3. 3. Dual-flow routes

    Why? The non-final route guarantees the full archive before any reduction.

    Add at the top: fw_archive (filter appname=='pan', pipeline passthru, out_s3_archive, Final off), then fw_siem (same filter, pipeline pl_fw_reduce to create, out_siem, Final). Send the data.
  4. 4. Parse and clean up

    Why? Empty and constant fields cost without adding anything.

    In pl_fw_reduce: Parser in Extract mode, type Key=Value on message; then Eval removing pad*, serial, vsys and _raw.
  5. 5. Drop, Suppress, Aggregations

    Why? Each function targets a different kind of waste.

    Add: Drop of allowed DNS; Suppress of identical denies (key source + port, 1 per 5 minutes); Aggregations of allowed SSL traffic per minute. Denies stay visible (with a counter).
  6. 6. Measure

    Why? The gain must be proven, and detections preserved.

    Resend the 2,000 lines. Compare in Monitoring › Data › Destinations the bytes of out_s3_archive and out_siem, check the archive in MinIO and that every denied source/port pair reached the SIEM.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 1 h

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Reducing, transforming and optimizing volumes with Cribl

Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...

Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search

Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...

Logs in Datadog: collection, pipelines, indexes and volume control

Collect logs with the agent, structure them with pipelines and processors (grok), use facets and ...

Cribl: the data pipeline for observability and security

Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...

Security with Datadog: Cloud SIEM, Cloud Security and application protection

Discover Datadog's security products: Cloud SIEM and detection rules, Cloud Security (posture, vu...

Elastic Security: SIEM, detection rules and investigation

Use Elastic Security as a SIEM: ECS data sources, detection rule types (query, EQL, ES|QL, thresh...

Reducing, masking and routing telemetry with Bindplane

Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...

Pipelines and functions: parse, enrich, mask

Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!