Final lab: Retention, VRL function and webhook alert with correction

What you will learn in this TP

• Set retention and indexes
• Write a VRL function
• Receive an alert

Introduction #

What you will learn in this section

• Set retention and indexes
• Write a VRL function
• Receive an alert

Context. Your prototype must become usable: keep logs 30 days, optimize searches and notify the team as soon as a payment fails.

Prerequisites: the first lab  ·  Estimated time: 45 min

A complete alert chain.

Setup #

Guided steps, with their solution.

1. 1. Prepare the stack

   Why? The receiver is a neighbouring container, so a private address: without this option OpenObserve refuses the destination. Do not reproduce in production.

   Add an echo container and the ZO_SKIP_SSRF_CHECKS option, then ingest the events of the first lab again.
   See the correction

   services:
     openobserve:
       image: public.ecr.aws/zinclabs/openobserve:latest
       ports:
         - "5080:5080"
       environment:
         ZO_DATA_DIR: /data
         ZO_ROOT_USER_EMAIL: root@example.com
         ZO_ROOT_USER_PASSWORD: Complexpass#123
         ZO_SKIP_SSRF_CHECKS: "true"
       volumes:
         - o2-data:/data
   
     echo:
       image: mendhak/http-https-echo:latest
   
   volumes:
     o2-data:

   curl -u 'root@example.com:Complexpass#123' \
     -H 'Content-Type: application/json' \
     http://localhost:5080/api/default/commandes/_json \
     -d '[
       {"level":"error","service":"paiement","message":"timeout vers la banque","duree_ms":3100},
       {"level":"info","service":"paiement","message":"ok","duree_ms":120},
       {"level":"info","service":"panier","message":"ok","duree_ms":40}
     ]'
   # {"code":200,"status":[{"name":"commandes","successful":3,"failed":0}]}

2. 2. Tune the stream

   Why? These settings decide cost and speed.

   Set 30 days of retention, full text on message, an index on service and a partition on level.
   See the correction

   curl -u 'root@example.com:Complexpass#123' -X PUT \
     -H 'Content-Type: application/json' \
     'http://localhost:5080/api/default/streams/commandes/settings?type=logs' \
     -d '{
       "data_retention": 30,
       "full_text_search_keys": {"add": ["message"]},
       "index_fields": {"add": ["service"]},
       "partition_keys": {"add": [{"field": "level", "types": "value"}]}
     }'

3. 3. Write a function

   Why? VRL transforms events without touching the application.

   Create the marque_grave function, then open it in the Functions menu and test it on an event.
   See the correction

   curl -u 'root@example.com:Complexpass#123' \
     -H 'Content-Type: application/json' \
     http://localhost:5080/api/default/functions \
     -d '{"name":"marque_grave","function":".grave = true\n.","params":"row","transType":0}'
   # {"code":200,"message":"Function saved successfully"}

4. 4. Create the alert chain

   Why? The alert needs the destination, which needs the template: order matters.

   Create the template, the destination, then the alert.
   See the correction

   A='root@example.com:Complexpass#123'
   curl -u "$A" -H 'Content-Type: application/json' \
     http://localhost:5080/api/default/alerts/templates \
     -d '{"name":"tpl_simple","type":"http","title":"",
          "body":"{\"alerte\":\"{alert_name}\",\"flux\":\"{stream_name}\",\"nb\":\"{alert_count}\"}"}'
   
   curl -u "$A" -H 'Content-Type: application/json' \
     http://localhost:5080/api/default/alerts/destinations \
     -d '{"name":"webhook_tp","url":"http://echo:8080/alerte","method":"post",
          "skip_tls_verify":false,"template":"tpl_simple","headers":{},"type":"http"}'

   A='root@example.com:Complexpass#123'
   curl -u "$A" -H 'Content-Type: application/json' \
     http://localhost:5080/api/v2/default/alerts \
     -d '{"name":"erreurs_paiement","stream_type":"logs","stream_name":"commandes",
          "is_real_time":false,"enabled":true,"owner":"root@example.com","folder_id":"default",
          "description":"Au moins une erreur en une minute","row_template":"","context_attributes":{},
          "query_condition":{"type":"sql","conditions":null,"promql":null,"promql_condition":null,
            "aggregation":null,"vrl_function":null,"multi_time_range":null,
            "sql":"SELECT count(*) AS nb FROM commandes WHERE level='"'"'error'"'"'"},
          "trigger_condition":{"period":1,"threshold":1,"frequency":1,"cron":"","operator":">=",
            "silence":0,"frequency_type":"minutes","timezone":null,"tolerance_in_seconds":null},
          "destinations":["webhook_tp"]}'

5. 5. Trigger the alert

   Why? An alert never fired is an untested alert.

   Send errors and read the logs of the echo container. Allow one to two minutes.
   See the correction

   for i in 1 2 3; do
     curl -s -u 'root@example.com:Complexpass#123' -H 'Content-Type: application/json' \
       http://localhost:5080/api/default/commandes/_json \
       -d '[{"level":"error","service":"paiement","message":"erreur de test"}]'
     sleep 30
   done
   docker compose logs echo | grep -i alerte

Difficulty level : ●●●○○ (3/5)