Logstash: role, pipeline concept and installation

1. Introduction #

The role of Logstash.

What Logstash is for #

Logstash is the processing engine of the Elastic Stack: it receives data from very varied sources, transforms it (parsing, cleaning, enrichment) and sends it to one or more destinations. It is written in Java and extended by more than 200 plugins.

• Centralise heterogeneous logs (syslog, files, databases, queues).
• Structure raw messages into ECS fields.
• Route to several destinations (Elasticsearch, S3, Kafka).

2. The pipeline concept #

Inputs, filters, outputs.

The three stages #

input  { beats { port => 5044 } }
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
output { elasticsearch { hosts => ["https://es:9200"] data_stream => true } }

Each event goes through the input, waits in the queue (memory by default), then a worker applies the filters in order and sends it to the outputs. Several workers process batches of events in parallel.

Good to know: Conditionals (if [field] == "value" { … }) apply filters or outputs to only some events.

3. Installation #

Packages, archive, Docker.

Installing #

# Debian / Ubuntu
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt update && sudo apt install logstash
sudo systemctl enable --now logstash

# Docker
docker run --rm -it -v $PWD/pipeline:/usr/share/logstash/pipeline docker.elastic.co/logstash/logstash:9.1.0

Good practice: Test the configuration before every restart: /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t.