What you will learn in this TP
- Configure an HEC source
- Extract fields with Regex Extract
- Enrich with Lookup
- Mask with Mask
- Test in the preview and wire the route
Introduction #
What you will learn in this section
- Configure an HEC source
- Extract fields with Regex Extract
- Enrich with Lookup
- Mask with Mask
- Test in the preview and wire the route
Context. The shop's access logs contain customer e-mail addresses in URLs. The SIEM must no longer receive them in clear, and analysts want a readable category for each HTTP code.
Prerequisites: a machine with Docker and Docker Compose (4 GB free memory), curl and jq, the Lab 1 instance · Estimated time: 50 min
A pipeline turning a raw Apache line into a structured, enriched event with no personal data in clear.Setup #
Guided steps, with their solution.
1. HEC source and data
Why? HEC is Splunk's HTTP protocol, widely used by applications.
Create a Splunk HEC sourcein_hecon port 8088 with the tokentp2-hec-token, then send 100 Apache lines.2. Capture a sample
Why? The pipeline is built on real lines.
Capture 100 events fromin_hecand save them as the sampleapache_access.log. Create thepl_apachepipeline and load the sample in the preview.3. Extract fields
Why? Structured fields allow filtering, aggregating and routing.
Add a Regex Extract function on_rawcreatingclient,method,uri,statusandbytes, then Numerify to convertstatusandbytesto numbers.4. Enrich
Why? A readable category saves every analyst from memorizing HTTP codes.
Create thehttp_status.csvtable in Processing › Knowledge › Lookups, then a Lookup function addingstatus_category.5. Mask e-mails
Why? The hash still allows counting distinct customers, without revealing their address.
Add a Mask function on_rawandurireplacing theemail=value with its SHA-256, then an Eval addingenv='prod'.6. Wire and check
Why? The preview proves the logic, the output proves the wiring.
Check in the preview (OUT) that no address remains in clear. Add anapacheroute (filtersourcetype=='apache:access', pipelinepl_apache, destinationout_files), resend data and check the files.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Questions about this lesson
No questions yet: be the first to ask for help or share your configuration.
Recommended Articles
Pipelines and functions: parse, enrich, mask
Building Cribl Stream pipelines: pre-processing, processing and post-processing, the function cat...
Advanced logs with OpenPipeline: processing, masking, buckets and retention
Manage logs in Dynatrace with OpenPipeline: ingestion, routing, DQL processing (parsing, enrichme...
Reducing, transforming and optimizing volumes with Cribl
Reducing the volume sent to SIEMs and observability tools without losing useful information: Drop...
Transforming with VRL and testing Vector pipelines
The VRL language (Vector Remap Language): parsing logs (JSON, regex, syslog), error handling, enr...
Sources, destinations and routes in Cribl Stream
Receiving and sending data with Cribl Stream: push, pull, collector and internal sources, event b...
Logs in Datadog: collection, pipelines, indexes and volume control
Collect logs with the agent, structure them with pipelines and processors (grok), use facets and ...
Reducing, masking and routing telemetry with Bindplane
Reduce ingestion costs with Bindplane: severity filters, deduplication, sampling, field removal, ...
Elasticsearch ingestion: Bulk API, ingest pipelines, enrich and reindex
Ingest and transform data in Elasticsearch: Bulk API, ingest pipelines and processors (grok, diss...
Cribl Edge: collecting close to the sources
Deploying Cribl Edge on Linux, Windows and Kubernetes: the agent's role, Fleets and Subfleets, co...