My first pipeline with correction

What you will learn in this TP

• Install Logstash
• Build a pipeline
• use the file input
• Combine different filters
• use the file output
• use conditions
• use variables

Introduction #

What you will learn in this section

• Install Logstash
• Build a pipeline
• use the file input
• Combine different filters
• use the file output
• use conditions
• use variables

This lab aims to teach you how to build a simple pipeline that will read a log file, process it and write its output to a new file

Prerequisites #

You must have a Linux machine with Logstash installed. If you do not have a machine or test environment, we invite you to do the lab on installing Logstash. For those doing the lab on the test environment, you can ask Vagrant to install Logstash:

export DEPLOY_LOGSTASH=true; vagrant up

Inputs configuration #

You must create a file in /etc/logstash/conf.d named "0-input-file-nginx-access.conf".

WARNING: In this lab, we specified the configuration file name (0-input-file-nginx-access.conf) but in a company, you will have to build your own naming standard that fits your context.

1. Reading a file

   In this file, you must add the logstash configuration to read the contents of the file "/var/log/rousseltm/nginx-access.log". If you use our Vagrant test environment, the log files will already be present on the VM in the /var/log/rousseltm folder. Otherwise, you will have to copy them manually: .
   You must add a type for this file with the value 'nginx_access'.

   Here is an example configuration to perform these actions:

Filters configuration #

You must create a second file named "100-filter-nginx-access.conf".

1. Data extraction

   You must identify the most suitable filter for a flat file and put the configuration that will extract the fields from the previously read log file knowing that this file has a single log format

   We choose dissect because it is the fastest plugin to read flat files and also because there is only one log format. After analysis we get this mapping for our dissect configuration:

   %{client.ip} - - [%{nginx.access.time}] "%{http.request.method} %{url.original} HTTP/%{http.version}" %{http.response.status_code} %{http.response.body.bytes} "%{http.request.referrer}" "%{user_agent.original}"

   As indicated during the course, do ECS as soon as possible.

2. Type modification

   We want to convert the "http_code" field to an integer. Indeed, there are often arithmetic operations on this field when making restitutions (dashboard, alerts...). For example, display all codes between 200 and 205.

   We will therefore use 'convert_datatype' to do it:

   convert_datatype => {
   	http_code => "int"
   }

3. Date adaptation

   As you noticed in the provided log file, the logs are not sorted. If you do not adapt the '@timestamp' field, you will have logs sorted by appearance in the file instead of sorting by generation date of the log line.

   We will therefore use 'date' to do it:

   date {
   	match => [ "nginx.access.time", "dd/MMM/yyyy:HH:mm:ss Z" ]
   	target => "@timestamp"
   }

4. Field deletion

   We want to remove the fields
   • event.original
   • message: as we have already extracted the information that interests us
   . Different methods must be used to remove them.

   WARNING: This is a good practice to save storage space and performance.

   In other labs, we will see that 'message' should only be removed in cases where we have no filter errors.

   We add a remove_field block and this gives us globally:

Outputs configuration #

You must create a file in /etc/logstash/conf.d named "200-output-file-nginx-access.conf".

1. Writing to a file

   In this file, you must add the logstash configuration so that it writes our processed log to the file "/var/log/rousseltm/parsed/TYPE.json"

   This configuration will allow you to write to the file:

Difficulty level : ●●○○○ (2/5)