RousselTMLEARNING

TP · Cribl: from discovery to expertise

Lab: halving the volume sent to the SIEM with correction

The correction is integrated into the TP (See the green boxes)

What you will learn in this TP
  • Deploy S3 storage and a fake SIEM
  • Route a dual flow
  • Drop the noise
  • Deduplicate and aggregate
  • Measure the reduction

Introduction #

What you will learn in this section
  • Deploy S3 storage and a fake SIEM
  • Route a dual flow
  • Drop the noise
  • Deduplicate and aggregate
  • Measure the reduction

Context. The SIEM licence is exceeded every month because of firewall logs. Security accepts filtering, provided everything can be recovered and every deny stays visible.

Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 60 min

A dual flow: full compressed archive and a SIEM flow reduced by at least 50%, measured in Monitoring.

Setup #

Guided steps, with their solution.

  1. 1. S3 storage and fake SIEM

    Why? MinIO plays S3, an echo server plays the SIEM.

    Create a shared network, MinIO with an archive bucket and an HTTP receiver siem.
    See the correction
    docker network create cribl-lab 2>/dev/null; docker network connect cribl-lab cribl
    docker run -d --name minio --network cribl-lab -p 9001:9001 \
      -e MINIO_ROOT_USER=cribl -e MINIO_ROOT_PASSWORD=cribl-lab-2026 \
      minio/minio:latest server /data --console-address :9001
    docker run --rm --network cribl-lab --entrypoint sh minio/mc:latest -c \
      'mc alias set m http://minio:9000 cribl cribl-lab-2026 && mc mb m/archive'
    docker run -d --name siem --network cribl-lab mendhak/http-https-echo:latest
  2. 2. Destinations

    Why? Two destinations with opposite roles: keep everything, or send only what is useful.

    Create out_s3_archive (Amazon S3: endpoint http://minio:9000, bucket archive, keys cribl / cribl-lab-2026, v4 signature, gzip JSON) and out_siem (Webhook: http://siem:8080/, JSON).
    See the correction
    Partitioning expression (out_s3_archive):
    `${C.Time.strftime(_time, '%Y/%m/%d/%H')}/${sourcetype}`
  3. 3. Dual-flow routes

    Why? The non-final route guarantees the full archive before any reduction.

    Add at the top: fw_archive (filter appname=='pan', pipeline passthru, out_s3_archive, Final off), then fw_siem (same filter, pipeline pl_fw_reduce to create, out_siem, Final). Send the data.
    See the correction
    cat > gen_fw.sh <<'EOF'
    #!/bin/sh
    # 2000 lignes de pare-feu au format clé=valeur, dont beaucoup de trafic DNS autorisé et de doublons
    for i in $(seq 1 2000); do
      case $((i % 10)) in
        0|1|2|3|4) l="action=allow src=10.0.$((i % 30)).$((i % 200)) dst=10.0.0.53 dport=53 proto=udp app=dns bytes=$((60 + i % 40))" ;;
        5|6)       l="action=allow src=10.0.$((i % 30)).$((i % 200)) dst=93.184.216.$((i % 5)) dport=443 proto=tcp app=ssl bytes=$((800 + i % 900))" ;;
        7|8)       l="action=deny src=203.0.113.$((i % 3)) dst=10.0.1.10 dport=22 proto=tcp app=ssh bytes=0" ;;
        9)         l="action=deny src=198.51.100.$((i % 50)) dst=10.0.1.20 dport=3389 proto=tcp app=rdp bytes=0" ;;
      esac
      echo "$l rule=r$((i % 4)) zone=dmz vsys=vsys1 serial=0123456789 pad1= pad2= pad3="
    done
    EOF
    chmod +x gen_fw.sh
    ./gen_fw.sh | while read -r l; do logger -n 127.0.0.1 -P 9514 -T -t pan "$l"; done
  4. 4. Parse and clean up

    Why? Empty and constant fields cost without adding anything.

    In pl_fw_reduce: Parser in Extract mode, type Key=Value on message; then Eval removing pad*, serial, vsys and _raw.
    See the correction
    Parser:  mode Extract · type Key=Value · source field message
    Eval:    remove fields  pad*, serial, vsys, _raw, message
  5. 5. Drop, Suppress, Aggregations

    Why? Each function targets a different kind of waste.

    Add: Drop of allowed DNS; Suppress of identical denies (key source + port, 1 per 5 minutes); Aggregations of allowed SSL traffic per minute. Denies stay visible (with a counter).
    See the correction
    Drop          filter: action=='allow' && dport==53
    Suppress      filter: action=='deny'   key: `${src}:${dport}`   allow 1 per 300 s
    Aggregations  filter: action=='allow' && app=='ssl'   window 60s
                  aggregates: count().as(flows), sum(bytes).as(bytes)   group by: src, dst, dport
  6. 6. Measure

    Why? The gain must be proven, and detections preserved.

    Resend the 2,000 lines. Compare in Monitoring › Data › Destinations the bytes of out_s3_archive and out_siem, check the archive in MinIO and that every denied source/port pair reached the SIEM.
    See the correction
    ./gen_fw.sh | while read -r l; do logger -n 127.0.0.1 -P 9514 -T -t pan "$l"; done; sleep 90
    docker run --rm --network cribl-lab --entrypoint sh minio/mc:latest -c \
      'mc alias set m http://minio:9000 cribl cribl-lab-2026 >/dev/null && mc du -r m/archive'
    docker logs siem 2>&1 | grep -o '"action":"deny"' | wc -l
    # expected: SIEM bytes ≤ 50 % of archive bytes; every deny src:dport present at least once

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 1 h

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Reducing, transforming and optimizing volumes with Cribl

Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...

Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search

Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...

Logs in Datadog: collection, pipelines, indexes and volume control

Collect logs with the agent, structure them with pipelines and processors (grok), use facets and ...

Cribl: the data pipeline for observability and security

Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...

Security with Datadog: Cloud SIEM, Cloud Security and application protection

Discover Datadog's security products: Cloud SIEM and detection rules, Cloud Security (posture, vu...

Elastic Security: SIEM, detection rules and investigation

Use Elastic Security as a SIEM: ECS data sources, detection rule types (query, EQL, ES|QL, thresh...

Reducing, masking and routing telemetry with Bindplane

Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...

Pipelines and functions: parse, enrich, mask

Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!