RousselTMLEARNING

TP · Cribl: from discovery to expertise

Lab: parsing, enriching and masking Apache logs with correction

The correction is integrated into the TP (See the green boxes)

What you will learn in this TP
  • Configure an HEC source
  • Extract fields with Regex Extract
  • Enrich with Lookup
  • Mask with Mask
  • Test in the preview and wire the route

Introduction #

What you will learn in this section
  • Configure an HEC source
  • Extract fields with Regex Extract
  • Enrich with Lookup
  • Mask with Mask
  • Test in the preview and wire the route

Context. The shop's access logs contain customer e-mail addresses in URLs. The SIEM must no longer receive them in clear, and analysts want a readable category for each HTTP code.

Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance  ·  Estimated time: 50 min

A pipeline turning a raw Apache line into a structured, enriched event with no personal data in clear.

Setup #

Guided steps, with their solution.

  1. 1. HEC source and data

    Why? HEC is Splunk's HTTP protocol, widely used by applications.

    Create a Splunk HEC source in_hec on port 8088 with the token tp2-hec-token, then send 100 Apache lines.
    See the correction
    HEC=http://localhost:8088/services/collector/event
    TOKEN='tp2-hec-token'
    for i in $(seq 1 100); do
      s=$(shuf -e 200 200 200 301 404 500 503 -n1)
      curl -s $HEC -H "Authorization: Splunk $TOKEN" -d "{\"sourcetype\":\"apache:access\",\"event\":\"10.1.2.$((i % 20)) - - [$(date '+%d/%b/%Y:%H:%M:%S %z')] \\\"GET /api/produit/$((i % 7))?email=client$i@exemple.fr HTTP/1.1\\\" $s $((RANDOM % 5000))\"}" >/dev/null
    done
  2. 2. Capture a sample

    Why? The pipeline is built on real lines.

    Capture 100 events from in_hec and save them as the sample apache_access.log. Create the pl_apache pipeline and load the sample in the preview.
    See the correction
    # Processing › Pipelines › Add Pipeline › pl_apache
    # Sample Data › Capture Data › filter: __inputId=='splunk_hec:in_hec' › Save as apache_access.log
  3. 3. Extract fields

    Why? Structured fields allow filtering, aggregating and routing.

    Add a Regex Extract function on _raw creating client, method, uri, status and bytes, then Numerify to convert status and bytes to numbers.
    See the correction
    /^(?<client>\S+) \S+ \S+ \[[^\]]+\] "(?<method>\S+) (?<uri>\S+) [^"]*" (?<status>\d{3}) (?<bytes>\d+)/
  4. 4. Enrich

    Why? A readable category saves every analyst from memorizing HTTP codes.

    Create the http_status.csv table in Processing › Knowledge › Lookups, then a Lookup function adding status_category.
    See the correction
    status,category
    200,succes
    301,redirection
    404,client
    500,serveur
    503,serveur
    # Lookup: file http_status.csv · lookup field status = event field status · output category → status_category
  5. 5. Mask e-mails

    Why? The hash still allows counting distinct customers, without revealing their address.

    Add a Mask function on _raw and uri replacing the email= value with its SHA-256, then an Eval adding env='prod'.
    See the correction
    Match regex:   /(email=)([^&\s]+)/g
    Replace expr:  `${g1}${C.Mask.sha256(g2)}`
    Eval:          env = 'prod'
  6. 6. Wire and check

    Why? The preview proves the logic, the output proves the wiring.

    Check in the preview (OUT) that no address remains in clear. Add an apache route (filter sourcetype=='apache:access', pipeline pl_apache, destination out_files), resend data and check the files.
    See the correction
    sleep 70
    zcat $(find $HOME/cribl-out -path '*apache*' -name '*.json.gz') | jq -c '{status, status_category, uri}' | head -5
    zcat $(find $HOME/cribl-out -path '*apache*' -name '*.json.gz') | grep -c '@exemple.fr' || echo 0

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 50 min

Sign in to track your progress and take notes. Sign in

💬 Questions about this lesson

No questions yet: be the first to ask for help or share your configuration.

Ask a question about this lesson →

Recommended Articles

Pipelines and functions: parse, enrich, mask

Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...

Advanced logs with OpenPipeline: processing, masking, buckets and retention

Manage logs in Dynatrace with OpenPipeline: ingestion, routing, DQL processing (parsing, enrichme...

Reducing, transforming and optimizing volumes with Cribl

Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...

Transforming with VRL and testing Vector pipelines

The VRL language (Vector Remap Language): parsing logs (JSON, regex, syslog), error handling, enr...

Sources, destinations and routes in Cribl Stream

Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...

Logs in Datadog: collection, pipelines, indexes and volume control

Collect logs with the agent, structure them with pipelines and processors (grok), use facets and ...

Reducing, masking and routing telemetry with Bindplane

Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...

Elasticsearch ingestion: Bulk API, ingest pipelines, enrich and reindex

Ingest and transform data in Elasticsearch: Bulk API, ingest pipelines and processors (grok, diss...

Cribl Edge: collecting close to the sources

Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!