Logstash in production: persistent queues, DLQ and multiple pipelines

1. Losing nothing: the persistent queue #

Memory or disk.

The problem #

By default, Logstash keeps in-flight events in an in-memory queue (queue.type: memory): a crash loses them. The persistent queue (PQ) writes them to disk between inputs and filters: they survive a restart and absorb spikes when Elasticsearch slows down.

Configuration #

# config/pipelines.yml
- pipeline.id: web
  path.config: /usr/share/logstash/pipeline/web.conf
  queue.type: persisted
  queue.max_bytes: 4gb          # au-delà : contre-pression vers les inputs
  path.queue: /var/lib/logstash/queue
  dead_letter_queue.enable: true

Common mistake: Believing the PQ protects against everything: it replaces neither reliable storage nor a sender able to retry. A UDP input (syslog) still loses whatever arrives during downtime; put Kafka in front if loss is unacceptable.

Good practice: Delivery is "at least once": after a recovery, an event may be indexed twice. Compute a stable ID (fingerprint filter) and pass it as document_id to make writes idempotent.

2. The dead letter queue #

Keeping rejections.

Principle #

When Elasticsearch rejects a document (400 or 404 error, typically a mapping conflict), the elasticsearch output writes it to the DLQ instead of dropping it, along with the reason. You then fix and replay it.

# Pipeline de reprise : lit la DLQ du pipeline « web »
input {
  dead_letter_queue {
    path => "/usr/share/logstash/data/dead_letter_queue"
    pipeline_id => "web"
    commit_offsets => true
  }
}
filter {
  # la raison du rejet : [@metadata][dead_letter_queue][reason]
  mutate { convert => { "status" => "integer" } }
}
output { elasticsearch { hosts => ["http://elasticsearch:9200"] data_stream => true } }

Good to know: Only "permanent" rejections go to the DLQ; transient errors (429, 503) are retried indefinitely by the output: backpressure then protects the data.

3. Multiple pipelines #

Isolating flows.

pipelines.yml #

A single large pipeline mixes flows: one slow filter or failing output blocks everything. pipelines.yml declares several independent pipelines, each with its own workers, queue and configuration.

- pipeline.id: entree
  config.string: |
    input { beats { port => 5044 } }
    output {
      if [event][dataset] == "nginx.access" { pipeline { send_to => ["web"] } }
      else if [event][category] == "authentication" { pipeline { send_to => ["securite"] } }
      else { pipeline { send_to => ["autres"] } }
    }
- pipeline.id: web
  path.config: /etc/logstash/conf.d/web.conf     # input { pipeline { address => "web" } }
  pipeline.workers: 4
- pipeline.id: securite
  path.config: /etc/logstash/conf.d/securite.conf
  queue.type: persisted

4. Performance #

Workers, batches, JVM.

Settings #

curl -s localhost:9600/_node/stats/pipelines?pretty | jq '.pipelines.web.events'
curl -s localhost:9600/_node/hot_threads?human=true | head -40

Good practice: Measure before tuning: compare events.in, events.out and per-filter duration_in_millis in _node/stats to find the costly filter.