Lab: Log pipeline, facets and volume control

What you will learn in this TP

• Create a grok pipeline
• Create facets and generated metrics
• Add an exclusion filter

Introduction #

What you will learn in this section

• Create a grok pipeline
• Create facets and generated metrics
• Add an exclusion filter

Context. An application's logs arrive as raw text and are expensive to index. You make them usable and reduce their cost.

Prerequisites: Lab 1 completed  ·  Estimated time: 45 min

Useful logs at a lower cost.

Hands-on #

Guided steps, with their solutions.

1. 1. Produce logs

   Why? A container that writes lines in a known format, including errors.

   Start a log generator tagged service:paiement.

2. 2. Create the pipeline

   Why? The grok parser turns the line into filterable attributes.

   In Logs > Pipelines, create a pipeline filtered on source:paiement with a grok parser, then a status remapper on the niveau attribute.

3. 3. Facets and measure

   Why? A facet filters, a measure aggregates.

   Create a facet on @resultat and a measure on @montant, then display the sum of declined amounts per minute in the explorer.

4. 4. Generated metric

   Why? A generated metric keeps working even for non-indexed logs.

   Create a paiement.refus metric (Generate Metrics) that counts @resultat:refuse logs, grouped by @motif.

5. 5. Exclusion filter

   Why? Accepted payments are numerous and rarely looked at.

   In the main index, add an exclusion filter that indexes only 10% of service:paiement @resultat:accepte logs. Check the drop in indexed volume in Usage.

Difficulty level : ●●●○○ (3/5)