Project 5: Getting hands-on with correction

What you will learn in this TP

• Create a 'drill down' between dashboards
• Create advanced (guided) variables: key-value
• Create advanced (custom) variables: key-value
• Identify and recognize the data source
• Build and use a Lucene query
• Use a Geomap visualization
• Customize the visualization
• Set the default time range
• Set a panel's title and description
• Build bar charts
• Set up a table with a traffic-light scheme
• Use value mapping
• Use regex
• Build pie charts
• Build a panel from another one
• Get familiar with the Logs visualization
• Run range queries with Lucene
• Build time series
• Use the Elasticsearch data source's Metrics mode
• Group data by a criterion
• Exclude a field by negation with Lucene
• Set up an alert

Introduction #

What you will learn in this section

• Create a 'drill down' between dashboards
• Create advanced (guided) variables: key-value
• Create advanced (custom) variables: key-value

For this project, you must have fully completed Project 4

1. Creating the dashboard

   In the folder you just created, create a dashboard named 'project 5' and assign it the tag 'project 5'. Then identify the correct data source that corresponds to Elasticsearch.

   To do this, in 'Dashboards', follow these steps:

   • Go to the folder created above
   • Click 'New' then 'New dashboard'

   In the window that opens, click '+ Add visualization'

2. Variable host_avance1

   You must create a variable named 'host_avance1' with the label 'Advanced host 1' that displays the list of domains (the 'host' field) present in an 'Infinity' data source. The URL to use to build the variable is: https://github.com/RousselTM/grafana-formation/blob/main/tp/data/host_avance1.json

   WARNING: notice that in the json there are two special fields, '__text' and '__value', which Grafana knows how to interpret as the variable's key and value.

   To do this, go to the dashboard's 'settings', 'Variables' tab, and click 'Add variable'. On the page that appears, you can use the default settings and set these values:

   • 'Variable type': select 'Query' since we're querying a URL.
   • 'Name' -> 'host'
   • 'Data source' -> select the Infinity data source
   • 'Query Type' -> Infinity
   • 'URL' -> 'https://raw.githubusercontent.com/RousselTM/grafana-formation/refs/heads/main/tp/data/variables1.json'. We converted the URL to RAW mode!

3. Variable host_avance2

   You must create a variable named 'host_avance2' with the label 'Advanced host 2' that displays the list of domains (the 'host' field) present in an 'Infinity' data source. The URL to use to build the variable is: https://github.com/RousselTM/grafana-formation/blob/main/tp/data/host_avance2.json

   WARNING: notice that in the json the two special fields '__text' and '__value' are gone. Also, the fields no longer have a title and are in key:value form. So you must use the 'Regex' field to extract the information we're interested in.

   You must add a new variable of type 'Query' and set the correct name and label. Select the 'Infinity' data source and give it the RAW-mode JSON URL. In Infinity, use 'Type' 'JSON' and in the 'Regex' field put this code:

   url:(?.*), title:(?.*)

   . To help build the regex, we recommend the site 'https://regex101.com/' using the 'Golang' flavor, which is the one used in Grafana.

4. Variable host_avance3

   You must create a variable named 'host_avance3' with the label 'Advanced host 3' that displays the list of domains (the 'host' field) present in an 'Infinity' data source. The URL to use to build the variable is: https://github.com/RousselTM/grafana-formation/blob/main/tp/data/host_avance3.json

   WARNING: a new use case. This time the information we're interested in is inside a sub-field. So you must switch from 'JSON' type to 'UQL' type. In the 'UQL' field that appears, you'll need to build the correct syntax.

   You must add a new variable of type 'Query' and set the correct name and label. Select the 'Infinity' data source and give it the RAW-mode JSON URL. In Infinity, use 'Type' 'UQL' and in the 'UQL' field put this code:

   parse-json
   | jsonata "formations.grafana.advanced.variables"
   | extend "__text"="data.key", "__value"="data.value"

5. Testing host_avanceX variables

   You must create 3 'Text' type visualizations placed side by side. Each will display the 3 following variable formats:

   Default: ${host_avanceX}
   Value: ${host_avanceX:value}
   Title: ${host_avanceX:text}

   To do this, add 3 visualizations with the 'Text' visualization type and use Grafana's options to resize and arrange them side by side. All that's left is to add the requested text in each visualization and customize the title and variable names in the 'Content' field.

Time series with Annotations #

What you will learn in this section

• Identify and recognize the data source
• Build and use a Lucene query
• Use a Geomap visualization
• Customize the visualization
• Set the default time range
• Set a panel's title and description

Get familiar with the map visualization

1. Data source

   You must identify and select the Elasticsearch data source.

   INFO: at work, you should define a naming convention in Grafana. A common mistake is putting the source type in the name, even though it's easily identifiable in the Grafana console

   To do this, choose the 'datasource3' data source, since it corresponds to Elasticsearch. To recognize it, the Elasticsearch logo is on the left of the name, and the type 'Elasticsearch' is on the right

2. The query

   You must build a Lucene query to display only documents with an HTTP status of '503' and whose 'machine.os' field starts with 'win'. As a reminder, you created a variable, so every query must include it!

   You should end up with this Lucene query:

   response:503 AND machine.os:win* AND host:$host

   . As a reminder, we asked earlier to add the filter on the variable, hence the addition of

   AND host:$host

3. Visualization type

   You must choose a visualization that displays data as a map and use these fields for coordinates:
   • Latitude: geo.coordinates.lat
   • Longitude: geo.coordinates.lon

   To do this, in 'Visualization', choose the 'Geomap' type, which displays a map. And in the settings:

   • [Map view] Initial view -> View: 'Fit to data', so the map zooms to the points.
   • [Map layers] Layer 1 -> Layer type: 'Markers', to show points as coordinate markers
   • [Map layers] Layer 1 -> Data: 'Query A', to select the query that will provide the information
   • [Map layers] Layer 1 -> Location Mode: 'Coords', to indicate how positions are identified: coordinates.
   • [Map layers] Layer 1 -> Latitude field: 'geo.coordinates.lat', to provide the latitude
   • [Map layers] Layer 1 -> Longitude field: 'geo.coordinates.lon', to provide the longitude

4. Title and description

   You must set the visualization's title to 'Geographic distribution of visitors' and description to 'Geographic representation of users'

   To do this, fill in the 'Title' and 'Description' fields in the 'Visualization' section

5. Point details

   You must configure the table to display only these fields when hovering over a point:
   • host
   • clientip
   • @timestamp
   • geo.dest
   • geo.src
   • message
   As well as any field needed for the map to work. The fields must be displayed in the exact order listed.

   You must use a transformation to do this. In the 'Transformations' tab, add the 'Organize fields by name' transformation and hide (by clicking the eye icon in front of the field name) every field that wasn't requested. Careful — we didn't ask to display the coordinates, but you can't hide them or the map will stop working! With the same transformation you can also reorder the fields as you like.

6. Default time range

   You must configure the dashboard to display data over 30 days by default.

   The simplest method is to select the period you want, in our case 'Last 30 days', then click 'Save dashboard'. In the window that opens, check 'Update default time range', enter a 'Message' to identify the changes made in case you need to revert to a previous version. Finally, click 'Save'.

Panel: SPC Histogram #

What you will learn in this section

• Build bar charts
• Set up a table with a traffic-light scheme
• Use value mapping
• Use regex

You must create a panel that displays a bar chart showing the breakdown of HTTP status codes. Set up a traffic-light color scheme:

• Green with 'OK': 2xx codes
• Yellow with 'Redirect': 3xx codes
• Orange with 'Client error': 4xx codes
• Red with 'Server error': 5xx codes

1. Grouping the data

   Using transformations, group the data by status code and count how many times each one appears over time.

   Use the 'Group by' transformation. The status code is stored in the 'response' field, so set 'Group by' on that field. All that's left is to apply the operation: 'Calculate' -> 'Count' on '@timestamp' (a unique field), for example

2. Converting the data

   A bar chart uses 'string' type fields. You must therefore convert the status codes to a string.

   Use the 'Convert field type' transformation. The status code is stored in the 'response' field, so set it to 'String'.

3. Coloring mode

   Tell Grafana to use the status code to color the bars.

   In the visualization settings, set 'response' (where the status code is stored) in the 'Color by field' option

4. Traffic-light scheme

   Set up the traffic-light scheme as requested above.

   You can't use 'Thresholds' since the field had to be converted to a string. So we'll use 'Value mappings' instead. Several options are possible, but the simplest is to use this Regex format:

   • condition: Regex, Regular expression: 2.*, Display text: OK, Color: green
   • condition: Regex, Regular expression: 3.*, Display text: Redirect, Color: yellow
   • condition: Regex, Regular expression: 4.*, Display text: Client error, Color: orange
   • condition: Regex, Regular expression: 5.*, Display text: Server error, Color: red

Panel: pie chart #

What you will learn in this section

• Build pie charts
• Build a panel from another one

You must create a panel that displays a bar chart showing the breakdown of HTTP status codes. Customize the labels according to this model:

• Label 'OK': 2xx code
• Label 'Redirect': 3xx code
• Label 'Client error': 4xx code
• Label 'Server error': 5xx code

1. Duplicating a panel

   Duplicate the panel you just created (Bar chart panel). Then resize them so they display on the same row. The bar chart should take up about 2/3 of the row and the clone 1/3.

   You must be in edit mode and click the 'Menu' (top right, shown as 3 dots) -> More ... -> Duplicate. Or hover over the panel and use the keyboard shortcut 'p d'. Once the panel is duplicated, resize and move them as needed

2. Changing visualization

   Change the visualization so the data is now displayed as pie slices.

   Use the 'Pie chart' visualization. That's not enough on its own — you also need to change the display mode (Show) to 'All values'

3. Traffic-light scheme

   Is it possible to apply a traffic-light color scheme with this type of visualization?

   No: some visualizations don't allow custom colors (and therefore traffic-light schemes): pie chart, timeseries...

Panel: Logs #

What you will learn in this section

• Get familiar with the Logs visualization
• Run range queries with Lucene

You must add a panel to the dashboard to display Logs.

1. HTTP errors

   Add a panel to display logs, showing only HTTP error logs.

   INFO: an HTTP error log is a log with an HTTP status code from 4xx to 5xx

   Add a new panel with 'Query type' set to 'Logs or Raw Data' and use this Lucene query

   response:[400 TO 599] AND host:$host

   . Finally, choose 'Logs' as the visualization. These settings are enough to start with

Panel: time series #

What you will learn in this section

• Build time series
• Use the Elasticsearch data source's Metrics mode
• Group data by a criterion
• Exclude a field by negation with Lucene

Build a time series with data from Elasticsearch. This graph should show how the number of documents evolves per HTTP status code

1. The query type

   Set the correct mode to build a time series. AND in the query, exclude all HTTP statuses in 2xx and 3xx

   Switch the query's 'Query type' to 'Metrics' mode and add this to 'Lucene Query'

   NOT response:[200 TO 3*] AND host:$host

2. The grouping

   Configure Grafana to group the metrics by time (in Elasticsearch, the default time field is '@timestamp') and by HTTP status code.

   Choose 'Metrics' as the 'Query type', and in the Metrics options set:

   • Metric: Count
   • Group By: 'Date Histogram' -> '@timestamp'
   • Group By: 'Terms' -> 'response.keyword'

   .

3. The visualization

   Select the visualization that displays time series.

   Select the 'Time series' visualization type.

Alerts #

What you will learn in this section

• Set up an alert

You must set up an alert on server errors.

INFO: a server error is a log with an HTTP status code of 5xx

1. Server error

   Set up an alert that triggers every time a server error occurs. Notifications must be sent to the 'RousselTM' contact point. You can check the notifications at: See notifications.

Difficulty level : ●●●○○ (3/5)