My second pipeline with correction

Prerequisites #

You must have done the lab 'My first pipeline'

Inputs configuration #

You must create a file in /etc/logstash/conf.d named "0-input-file-advanced.conf".

1. Reading a file

   In this file, you must add the logstash configuration to read the contents of all log files starting with: "/var/log/rousseltm/advanced-".
   
   You must add a type for this file with the value 'advanced'.

   This example configuration will allow you to read this file:

   root@logstash-formation:/#cat /etc/logstash/conf.d/0-input-file-advanced.conf
   input {
   	file {
   		path => "/var/log/rousseltm/advanced-*.log"
   		"type" => "advanced"
   		codec => multiline {
   			pattern => "^\[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\] "
   			negate => true
   			what => "previous"
   		}
   	}
   }
   

   If you look closely at the file, you will see that from line 1 to line 8 it is the same log line. You must therefore indicate to Logstash that it is the same line. We therefore use a multiline codec to indicate how to identify each line start. The 'pattern' parameter supports grok.

Filters configuration #

You must create a second file named "101-filter-advances.conf".

1. Data extraction

   You must identify the most suitable filter for a flat file and put the configuration that will extract the fields from the previously read log files knowing that these files are made up of several log formats.

   We will use a grok filter as there are several log formats in the same file. We will therefore provide several patterns for the grok configuration.
   
   Furthermore, we will add an 'if' condition so that this new filter only processes data of type 'advance'. You will therefore need to review your simple filter to do the same.

2. Type modification

   We want to convert the "HTTP_CODE" field to an integer. Indeed, there are often arithmetic operations on this field when making restitutions (dashboard, alerts...). For example, display all codes between 200 and 205.

   We add a mutate filter with the convert option to convert the field to an integer:

   root@logstash-formation:/# cat /etc/logstash/conf.d/101-filter-advanced.conf
   ...
   	mutate {
   		convert => {"http_code" => "integer" }
   	}
   ...

3. Field deletion

   We will remove the message field as we have already extracted the information that interests us.

   We add a remove_field block and this gives us globally:

   root@logstash-formation:/# cat /etc/logstash/conf.d/101-filter-advanced.conf
   filter {
   	if [type] == "advanced" {
   		grok {
   			match => {
   				"message" => [
   					"\[(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\] %{DATA} \[%{NUMBER:http_code}\]: %{WORD:http_verb} %{URIPATHPARAM:http_request}%{GREEDYDATA:others}",
   					"\[(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\] %{DATA} %{WORD:status}"
   				]
   			}
   			remove_field => ["message"]
   		}
   		mutate {
   			convert => { "http_code" => "integer" }
   		}
   
   	}
   
   }

Outputs configuration #

You must modify the file "200-output-file-nginx-access".

1. Writing to a file

   In this file, you must update the logstash configuration so that it writes the logs to the file that corresponds to its 'type'

   This configuration will allow you to write to the file dynamically based on the log type:

   root@logstash-formation:/# cat 200-output-file
   output {
   	file {
   		path => "/var/log/rousseltm/parsed/%{[type]}.json"
   	}
   }

Difficulty level : ●●●○○ (3/5)