Lab: Collecting and Shipping Logs to Loki with correction

What you will learn in this TP

• Read log files
• Extract and add labels
• Send to Loki and check

Introduction #

What you will learn in this section

• Read log files
• Extract and add labels
• Send to Loki and check

Context. An application writes JSON logs to files, and nobody reads them before an incident. You centralize them in Loki.

Prerequisites: Docker Compose, curl  ·  Estimated time: 30 min

Logs searchable by level and environment.

Log pipeline #

Guided steps, with their solution.

1. 1. Prepare the stack

   Why? The volume keeps read positions: without it, Alloy would re-read everything at each restart.

   Write a Compose file with Loki and Alloy.
   See the correction

   services:
     loki:
       image: grafana/loki:latest
       command: -config.file=/etc/loki/local-config.yaml
       ports:
         - "3100:3100"
   
     alloy:
       image: grafana/alloy:v1.21.3
       command: run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
       ports:
         - "12345:12345"
       volumes:
         - ./config.alloy:/etc/alloy/config.alloy:ro
         - ./logs:/var/log/myapp:ro
         - alloy-data:/var/lib/alloy/data
       depends_on: [loki]
   
   volumes:
     alloy-data:

2. 2. Read the files

   Why? local.file_match finds files, loki.source.file reads them and remembers its position.

   In config.alloy, read /var/log/myapp/*.log with the job=myapp label.
   See the correction

   local.file_match "app" {
     path_targets = [{ "__path__" = "/var/log/myapp/*.log", "job" = "myapp" }]
   }
   
   loki.source.file "app_logs" {
     targets    = local.file_match.app.targets
     forward_to = [loki.process.app.receiver]
   }

3. 3. Process the lines

   Why? stage.labels takes an extracted value; a fixed value needs stage.static_labels.

   Extract level from the JSON to make it a label, add env=production and keep msg as text.
   See the correction

   loki.process "app" {
     forward_to = [loki.write.local_loki.receiver]
   
     stage.json {
       expressions = { level = "", msg = "" }
     }
   
     stage.labels {
       values = { level = "" }
     }
   
     stage.static_labels {
       values = { env = "production" }
     }
   
     stage.output {
       source = "msg"
     }
   }

4. 4. Send to Loki

   Why? In the Compose network, Loki is reached by its service name.

   Add the sending component.
   See the correction

   loki.write "local_loki" {
     endpoint {
       url = "http://loki:3100/loki/api/v1/push"
     }
   }

5. 5. Check

   Why? A healthy component does not prove that data arrives.

   Create a log file, start the stack and query Loki: you should see the env and level labels, and the texts paiement refusé and commande créée.
   See the correction

   mkdir -p logs
   cat >> logs/app.log <<'EOF'
   {"level":"error","msg":"paiement refusé"}
   {"level":"info","msg":"commande créée"}
   EOF
   docker compose up -d
   sleep 20
   curl -s http://localhost:3100/loki/api/v1/labels
   curl -s -G http://localhost:3100/loki/api/v1/query_range \
     --data-urlencode 'query={job="myapp",env="production"}' \
     --data-urlencode "start=$(( $(date +%s)-600 ))000000000"

Difficulty level : ●●○○○ (2/5)