Introduction

What you will learn in this section :

• Install Logstash
• Build a pipeline
• use the file input
• Combine different filters
• use the file output
• use conditions
• use variables

Prerequisites

export DEPLOY_LOGSTASH=true; vagrant up

Inputs configuration

WARNING: In this lab, we specified the configuration file name (0-input-file-nginx-access.conf) but in a company, you will have to build your own naming standard that fits your context.

1. Reading a file

   In this file, you must add the logstash configuration to read the contents of the file "/var/log/rousseltm/nginx-access.log". If you use our Vagrant test environment, the log files will already be present on the VM in the /var/log/rousseltm folder. Otherwise, you will have to copy them manually: .
   You must add a type for this file with the value 'nginx_access'.

Filters configuration

1. Data extraction

   You must identify the most suitable filter for a flat file and put the configuration that will extract the fields from the previously read log file knowing that this file has a single log format

2. Type modification

   We want to convert the "http_code" field to an integer. Indeed, there are often arithmetic operations on this field when making restitutions (dashboard, alerts...). For example, display all codes between 200 and 205.

3. Date adaptation

   As you noticed in the provided log file, the logs are not sorted. If you do not adapt the '@timestamp' field, you will have logs sorted by appearance in the file instead of sorting by generation date of the log line.

4. Field deletion

   We want to remove the fields
   • event.original
   • message: as we have already extracted the information that interests us
   . Different methods must be used to remove them.

   WARNING: This is a good practice to save storage space and performance.

   In other labs, we will see that 'message' should only be removed in cases where we have no filter errors.

Outputs configuration

1. Writing to a file

   In this file, you must add the logstash configuration so that it writes our processed log to the file "/var/log/rousseltm/parsed/TYPE.json"