Lab: Parse nginx logs with Vector and VRL

What you will learn in this TP

• Write a VRL transformation
• Route to Loki
• Produce metrics

Introduction #

What you will learn in this section

• Write a VRL transformation
• Route to Loki
• Produce metrics

Context. The website exposes no HTTP metrics, but nginx writes access logs. You get both structured logs and metrics out of them.

Prerequisites: Docker Compose  ·  Estimated time: 45 min

Structured logs and metrics from a simple file.

Setup #

Guided steps, with their solutions.

1. 1. Prepare nginx and the traffic

   Why? nginx writes its logs to a volume shared with Vector.

   Create a Compose file with nginx (logs in a volume), a request generator (including 404s) and Loki.

2. 2. Write the transformation

   Why? VRL extracts the fields and converts the types: you can then filter and aggregate.

   Create vector.yaml with a file source and a remap that parses the combined format.

3. 3. Run and observe

   Why? vector tap shows the transformed events live.

   Add Vector to the Compose file, start it and inspect the output of parse.

4. 4. Derive metrics

   Why? To count requests by status without touching nginx.

   Add a log_to_metric and a prometheus_exporter sink on port 9598, then read the metrics.

5. 5. Check in Loki

   Why? The extracted fields can be filtered directly in LogQL.

   Display the 404 requests.

Difficulty level : ●●●○○ (3/5)