1. Inputs #
Receiving data.
Common inputs #
| Input | Use | Key option |
|---|---|---|
| beats / elastic_agent | receive Beats and Elastic Agent | port => 5044, ssl_enabled |
| file | read local files | path, start_position, sincedb_path |
| tcp / udp / syslog | network streams | port, codec |
| http | receive webhooks | port, codec => json |
| kafka | consume topics | bootstrap_servers, topics, group_id |
| jdbc | query an SQL database | statement, schedule, tracking_column |
input {
beats { port => 5044 }
file { path => "/var/log/app/*.log" start_position => "beginning" }
jdbc {
jdbc_connection_string => "jdbc:postgresql://db:5432/app"
jdbc_user => "lecture"
jdbc_driver_class => "org.postgresql.Driver"
schedule => "*/5 * * * *"
statement => "SELECT * FROM commandes WHERE maj > :sql_last_value"
use_column_value => true
tracking_column => "maj"
tracking_column_type => "timestamp"
}
}Good to know: The codec decodes data at input (json,multiline,plain) or encodes it at output.
2. Outputs #
Sending data.
The elasticsearch output #
output {
elasticsearch {
hosts => ["https://es1:9200", "https://es2:9200"]
api_key => "${ES_API_KEY}"
ssl_certificate_authorities => ["/etc/logstash/ca.crt"]
data_stream => true
data_stream_type => "logs"
data_stream_dataset => "app"
data_stream_namespace => "prod"
}
}Good practice: Write to data streams (logs-app-prod) rather than dated indices: rollover, ILM retention and index templates are then handled by Elasticsearch.Other outputs #
| Output | Use |
|---|---|
| stdout { codec => rubydebug } | debugging |
| file | local copy |
| s3 | cheap long-term archiving |
| kafka | feed other consumers |
| pipeline | send to another Logstash pipeline |
Common mistake: Putting the Elasticsearch password in clear text in the pipeline file: use the keystore (bin/logstash-keystore add ES_API_KEY) and reference the value with${ES_API_KEY}.
