Lab: Logstash in production (persistent queue, DLQ, multiple pipelines)

What you will learn in this TP

• Declare several pipelines
• Enable the persistent queue and the DLQ
• Trigger and read a rejection
• Monitor Logstash

Introduction #

What you will learn in this section

• Declare several pipelines
• Enable the persistent queue and the DLQ
• Trigger and read a rejection
• Monitor Logstash

Context. Documents vanish between Logstash and Elasticsearch. You set up a persistent queue and a DLQ to lose nothing, and learn to find rejected events.

Prerequisites: Docker and Docker Compose, 4 GB of free RAM, jq  ·  Estimated time: 45 min

Hands-on #

Guided steps, with their solution.

1. 1. Lab files

   Why? The "web" pipeline writes to the logs-web-lab data stream; the "dlq" pipeline reads its rejections.

2. 2. A strict mapping

   Why? The status field is declared as an integer: a non-numeric status will be rejected by Elasticsearch, feeding the DLQ.

3. 3. Send events

   Send three valid events and one whose status is not numeric.

4. 4. Read the DLQ

   Why? The rejected document is not lost: the "dlq" pipeline shows it with the rejection reason.

5. 5. Monitor and test the queue

   Check the pipeline statistics, stop Elasticsearch, send events, then restart it: they are delivered.

Difficulty level : ●●●○○ (3/5)