What you will learn in this TP
- Configure an HEC source
- Extract fields with Regex Extract
- Enrich with Lookup
- Mask with Mask
- Test in the preview and wire the route
Introduction #
What you will learn in this section
- Configure an HEC source
- Extract fields with Regex Extract
- Enrich with Lookup
- Mask with Mask
- Test in the preview and wire the route
Context. The shop's access logs contain customer e-mail addresses in URLs. The SIEM must no longer receive them in clear, and analysts want a readable category for each HTTP code.
Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance · Estimated time: 50 min
A pipeline turning a raw Apache line into a structured, enriched event with no personal data in clear.Setup #
Guided steps, with their solution.
1. HEC source and data
Why? HEC is Splunk's HTTP protocol, widely used by applications.
Create a Splunk HEC sourcein_hecon port 8088 with the tokentp2-hec-token, then send 100 Apache lines.2. Capture a sample
Why? The pipeline is built on real lines.
Capture 100 events fromin_hecand save them as the sampleapache_access.log. Create thepl_apachepipeline and load the sample in the preview.3. Extract fields
Why? Structured fields allow filtering, aggregating and routing.
Add a Regex Extract function on_rawcreatingclient,method,uri,statusandbytes, then Numerify to convertstatusandbytesto numbers.4. Enrich
Why? A readable category saves every analyst from memorizing HTTP codes.
Create thehttp_status.csvtable in Processing › Knowledge › Lookups, then a Lookup function addingstatus_category.5. Mask e-mails
Why? The hash still allows counting distinct customers, without revealing their address.
Add a Mask function on_rawandurireplacing theemail=value with its SHA-256, then an Eval addingenv='prod'.6. Wire and check
Why? The preview proves the logic, the output proves the wiring.
Check in the preview (OUT) that no address remains in clear. Add anapacheroute (filtersourcetype=='apache:access', pipelinepl_apache, destinationout_files), resend data and check the files.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Dynatrace: GDPR and data privacy
Learn how to configure Dynatrace to comply with the GDPR, mask sensitive data and enable the coll...
Grafana Alloy: Log Management with Loki
Discover how to configure Grafana Alloy to read log files, journald, or network streams, process ...
Cribl: the data pipeline for observability and security
Discover Cribl: why a telemetry data pipeline, the Cribl suite (Stream, Edge, Lake, Search, Cribl...
Advanced Datadog logs: indexes, exclusion filters, Flex Logs, archives and Sensitive Data Scanner
Control the cost and compliance of Datadog logs: pipelines and processors, indexes and retention,...
Bindplane: managing your OpenTelemetry telemetry pipelines
Understand Bindplane: a control plane for OpenTelemetry collectors via OpAMP, BDOT Collector, con...
OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines
Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...
Logs with OpenTelemetry: data model, appenders, correlation and collection
Master the log signal: the LogRecord data model, the Logs Bridge API and appenders (Logback, Log4...
Archiving, replaying and searching: S3, Replay, Cribl Lake and Cribl Search
Keeping all data cheaply and finding it on demand: partitioned S3 destination, JSON and Parquet f...
Cribl Stream in production: distributed architecture, persistent queues and monitoring
Taking Cribl Stream to production: Leader, Worker Groups and processes, commit and deploy, scalin...