Project 4: Working with Elasticsearch with correction

What you will learn in this TP

• Get familiar with the Grafana interface
• Creating dashboards
• Assigning tags to a dashboard
• Identify and recognize the data source
• Build and use a Lucene query
• Use a Geomap visualization
• Customize the visualization
• Set the default time range
• Set a panel's title and description
• Build bar charts
• Set up a table with a traffic-light scheme
• Use value mapping
• Use regex
• Build pie charts
• Build a panel from another one
• Get familiar with the Logs visualization
• Run range queries with Lucene
• Build time series
• Use the Elasticsearch data source's Metrics mode
• Group data by a criterion
• Exclude a field by negation with Lucene
• Set up an alert

Introduction #

What you will learn in this section

• Get familiar with the Grafana interface
• Creating dashboards
• Assigning tags to a dashboard

To complete this project, you must install your own Grafana instance (a lab is available for that). But if you're attending an in-person course, you can ask your instructor for credentials to connect to the instance at https://demo.rousseltm.fr

NOTICE: don't hesitate to ask us questions, because there are concepts from other products that you need to know (or master) to easily configure your Grafana dashboards:

• Query type: we recommend watching this video, which explains the different types of queries you can run in Grafana with Elasticsearch. Here's the summary needed for this lab:
  • Metrics: used to build time series charts. Relies on Elasticsearch aggregations (date_histogram, terms, avg, count, etc.).
  • Logs (default value): observability/logs-oriented mode. Displays events (documents) chronologically. Grafana formats fields (@timestamp, message, log.level, etc.) to look like a log explorer (like Kibana Discover).
  • Raw data: returns Elasticsearch documents as they are stored (raw JSON). Less user-friendly than Logs mode, but closer to the raw data.
  • Raw Document (deprecated): older version of Raw data. Used _source directly and didn't offer Grafana's more recent optimizations.


• Lucene query: we recommend watching this video, which explains the difference between Elasticsearch's query languages. You'll need to understand the Lucene syntax for this lab, and the essentials are explained in the video

.

1. Getting started

   You must create a folder and place your dashboard in it. If this is a group course, the instructor will assign you a dedicated folder.

   To do this, in 'Dashboards', follow these steps:

   • Go to the 'grafana-participants' folder
   • Click 'New' then 'New folder'

2. Creating the dashboard

   In the folder you just created, create a dashboard named 'project 4' and assign it the tag 'project 4'. Then identify the correct data source that corresponds to Elasticsearch.

   To do this, in 'Dashboards', follow these steps:

   • Go to the folder created above
   • Click 'New' then 'New dashboard'

   In the window that opens, click '+ Add visualization'

3. Creating a variable

   You must create a variable named 'host' that displays the list of domains (the 'host' field) present in the Elasticsearch data source. You must allow selecting multiple options in the variable.

   WARNING: each of the panels below must vary depending on the chosen variable

   To do this, go to the dashboard's 'settings', 'Variables' tab, and click 'Add variable'. On the page that appears, you can use the default settings and set these values:

   • 'Variable type': select 'Query' since we're querying an Elasticsearch index.
   • 'Name' -> 'host'
   • 'Data source' -> select the Elasticsearch data source
   • 'Query' -> ' {"find": "terms", "field": "host.keyword"} '
   • Check the 'Multi-value' option in the 'Selection options' section to allow selecting multiple values

Panel: map #

What you will learn in this section

• Identify and recognize the data source
• Build and use a Lucene query
• Use a Geomap visualization
• Customize the visualization
• Set the default time range
• Set a panel's title and description

Get familiar with the map visualization

1. Data source

   You must identify and select the Elasticsearch data source.

   INFO: at work, you should define a naming convention in Grafana. A common mistake is putting the source type in the name, even though it's easily identifiable in the Grafana console

   To do this, choose the 'datasource3' data source, since it corresponds to Elasticsearch. To recognize it, the Elasticsearch logo is on the left of the name, and the type 'Elasticsearch' is on the right

2. The query

   You must build a Lucene query to display only documents with an HTTP status of '503' and whose 'machine.os' field starts with 'win'. As a reminder, you created a variable, so every query must include it!

   You should end up with this Lucene query:

   response:503 AND machine.os:win* AND host:$host

   . As a reminder, we asked earlier to add the filter on the variable, hence the addition of

   AND host:$host

3. Visualization type

   You must choose a visualization that displays data as a map and use these fields for coordinates:
   • Latitude: geo.coordinates.lat
   • Longitude: geo.coordinates.lon

   To do this, in 'Visualization', choose the 'Geomap' type, which displays a map. And in the settings:

   • [Map view] Initial view -> View: 'Fit to data', so the map zooms to the points.
   • [Map layers] Layer 1 -> Layer type: 'Markers', to show points as coordinate markers
   • [Map layers] Layer 1 -> Data: 'Query A', to select the query that will provide the information
   • [Map layers] Layer 1 -> Location Mode: 'Coords', to indicate how positions are identified: coordinates.
   • [Map layers] Layer 1 -> Latitude field: 'geo.coordinates.lat', to provide the latitude
   • [Map layers] Layer 1 -> Longitude field: 'geo.coordinates.lon', to provide the longitude

4. Title and description

   You must set the visualization's title to 'Geographic distribution of visitors' and description to 'Geographic representation of users'

   To do this, fill in the 'Title' and 'Description' fields in the 'Visualization' section

5. Point details

   You must configure the table to display only these fields when hovering over a point:
   • host
   • clientip
   • @timestamp
   • geo.dest
   • geo.src
   • message
   As well as any field needed for the map to work. The fields must be displayed in the exact order listed.

   You must use a transformation to do this. In the 'Transformations' tab, add the 'Organize fields by name' transformation and hide (by clicking the eye icon in front of the field name) every field that wasn't requested. Careful — we didn't ask to display the coordinates, but you can't hide them or the map will stop working! With the same transformation you can also reorder the fields as you like.

6. Default time range

   You must configure the dashboard to display data over 30 days by default.

   The simplest method is to select the period you want, in our case 'Last 30 days', then click 'Save dashboard'. In the window that opens, check 'Update default time range', enter a 'Message' to identify the changes made in case you need to revert to a previous version. Finally, click 'Save'.

Panel: bar gauge chart #

What you will learn in this section

• Build bar charts
• Set up a table with a traffic-light scheme
• Use value mapping
• Use regex

You must create a panel that displays a bar chart showing the breakdown of HTTP status codes. Set up a traffic-light color scheme:

• Green with 'OK': 2xx codes
• Yellow with 'Redirect': 3xx codes
• Orange with 'Client error': 4xx codes
• Red with 'Server error': 5xx codes

1. Grouping the data

   Using transformations, group the data by status code and count how many times each one appears over time.

   Use the 'Group by' transformation. The status code is stored in the 'response' field, so set 'Group by' on that field. All that's left is to apply the operation: 'Calculate' -> 'Count' on '@timestamp' (a unique field), for example

2. Converting the data

   A bar chart uses 'string' type fields. You must therefore convert the status codes to a string.

   Use the 'Convert field type' transformation. The status code is stored in the 'response' field, so set it to 'String'.

3. Coloring mode

   Tell Grafana to use the status code to color the bars.

   In the visualization settings, set 'response' (where the status code is stored) in the 'Color by field' option

4. Traffic-light scheme

   Set up the traffic-light scheme as requested above.

   You can't use 'Thresholds' since the field had to be converted to a string. So we'll use 'Value mappings' instead. Several options are possible, but the simplest is to use this Regex format:

   • condition: Regex, Regular expression: 2.*, Display text: OK, Color: green
   • condition: Regex, Regular expression: 3.*, Display text: Redirect, Color: yellow
   • condition: Regex, Regular expression: 4.*, Display text: Client error, Color: orange
   • condition: Regex, Regular expression: 5.*, Display text: Server error, Color: red

Panel: pie chart #

What you will learn in this section

• Build pie charts
• Build a panel from another one

You must create a panel that displays a bar chart showing the breakdown of HTTP status codes. Customize the labels according to this model:

• Label 'OK': 2xx code
• Label 'Redirect': 3xx code
• Label 'Client error': 4xx code
• Label 'Server error': 5xx code

1. Duplicating a panel

   Duplicate the panel you just created (Bar chart panel). Then resize them so they display on the same row. The bar chart should take up about 2/3 of the row and the clone 1/3.

   You must be in edit mode and click the 'Menu' (top right, shown as 3 dots) -> More ... -> Duplicate. Or hover over the panel and use the keyboard shortcut 'p d'. Once the panel is duplicated, resize and move them as needed

2. Changing visualization

   Change the visualization so the data is now displayed as pie slices.

   Use the 'Pie chart' visualization. That's not enough on its own — you also need to change the display mode (Show) to 'All values'

3. Traffic-light scheme

   Is it possible to apply a traffic-light color scheme with this type of visualization?

   No: some visualizations don't allow custom colors (and therefore traffic-light schemes): pie chart, timeseries...

Panel: Logs #

What you will learn in this section

• Get familiar with the Logs visualization
• Run range queries with Lucene

You must add a panel to the dashboard to display Logs.

1. HTTP errors

   Add a panel to display logs, showing only HTTP error logs.

   INFO: an HTTP error log is a log with an HTTP status code from 4xx to 5xx

   Add a new panel with 'Query type' set to 'Logs or Raw Data' and use this Lucene query

   response:[400 TO 599] AND host:$host

   . Finally, choose 'Logs' as the visualization. These settings are enough to start with

Panel: time series #

What you will learn in this section

• Build time series
• Use the Elasticsearch data source's Metrics mode
• Group data by a criterion
• Exclude a field by negation with Lucene

Build a time series with data from Elasticsearch. This graph should show how the number of documents evolves per HTTP status code

1. The query type

   Set the correct mode to build a time series. AND in the query, exclude all HTTP statuses in 2xx and 3xx

   Switch the query's 'Query type' to 'Metrics' mode and add this to 'Lucene Query'

   NOT response:[200 TO 3*] AND host:$host

2. The grouping

   Configure Grafana to group the metrics by time (in Elasticsearch, the default time field is '@timestamp') and by HTTP status code.

   Choose 'Metrics' as the 'Query type', and in the Metrics options set:

   • Metric: Count
   • Group By: 'Date Histogram' -> '@timestamp'
   • Group By: 'Terms' -> 'response.keyword'

   .

3. The visualization

   Select the visualization that displays time series.

   Select the 'Time series' visualization type.

Alerts #

What you will learn in this section

• Set up an alert

You must set up an alert on server errors.

INFO: a server error is a log with an HTTP status code of 5xx

1. Server error

   Set up an alert that triggers every time a server error occurs. Notifications must be sent to the 'RousselTM' contact point. You can check the notifications at: See notifications.

Difficulty level : ●●●○○ (3/5)