Lab: Ingest pipeline and Data Prepper

What you will learn in this TP

• Create and simulate a pipeline
• Deploy Data Prepper
• Type fields correctly

Introduction #

What you will learn in this section

• Create and simulate a pipeline
• Deploy Data Prepper
• Type fields correctly

Context. Your logs arrive poorly structured. You normalize them first inside the cluster, then with a dedicated collector.

Prerequisites: the previous lab  ·  Estimated time: 45 min

Clean, well-typed data.

Setup #

Guided steps, with their solution.

1. 1. Create and simulate a pipeline

   Why? _simulate avoids damaging data.

   Create the pipeline, then test it without indexing anything.

2. 2. Apply it

   Why? The pipeline is chosen at write time.

   Reindex the batch of the previous lab with the pipeline, and find the document of user alice.

3. 3. Prepare the typing

   Why? Without a template, clientip would be text and network search would fail.

   Create the web-logs template.

4. 4. Deploy Data Prepper

   Why? Processing leaves the cluster and relieves its nodes.

   Write pipelines.yaml, add the service and start it.

5. 5. Send and check

   Why? Network search proves that the ip type is in place.

   Send two log lines, wait a few seconds, then search addresses in 10.0.0.0/8.

Difficulty level : ●●○○○ (2/5)