2. Processing pipelines #
Filtering, parsing (JSON, Regex), and relabeling.
Processing line by line #
loki.process applies stages, in order. They share a working area, the “extracted map”: one stage drops values there, a later one uses them.
loki.process "app" {
forward_to = [loki.write.local.receiver]
stage.json {
expressions = { level = "", service = "", trace_id = "", ts = "", msg = "" }
}
stage.drop {
source = "level"
value = "debug"
}
stage.timestamp {
source = "ts"
format = "RFC3339"
}
stage.labels {
values = { level = "", service = "" }
}
stage.structured_metadata {
values = { trace_id = "" }
}
stage.static_labels {
values = { env = "production" }
}
stage.output {
source = "msg"
}
}
loki.write "local" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}What this pipeline produces #
Tested end to end with a real Loki, on this file:
{"ts":"2026-11-05T10:00:00Z","level":"error","service":"paiement","trace_id":"abc123","msg":"timeout banque"}
{"ts":"2026-11-05T10:00:00Z","level":"debug","service":"paiement","trace_id":"abc124","msg":"détail inutile"}
{"ts":"2026-11-05T10:00:00Z","level":"info","service":"panier","trace_id":"abc125","msg":"article ajouté"}
ligne non json| Line | In Loki |
|---|---|
error, paiement | Text timeout banque, labels level=error, service=paiement, env=production, metadata trace_id=abc123 |
debug | Discarded: loki_process_dropped_lines_total is 1 |
info, panier | Text article ajouté, labels level=info, service=panier |
ligne non json | Kept as is, with no level or service |
The most useful stages #
| Stage | Purpose |
|---|---|
stage.json, stage.regex, stage.logfmt | Extract values |
stage.labels | Turn an extracted value into a label |
stage.static_labels | Add a label with a fixed value |
stage.structured_metadata | Keep a varied value without creating streams |
stage.timestamp | Use the log time |
stage.drop | Discard lines |
stage.output | Replace the line text |
stage.multiline | Group several lines |
Do not confusestage.labelsandstage.static_labels. Instage.labels { values = { env = "production" } },productionis the name of an extracted value, not text.alloy validateaccepts it, but we checked that the label never appears. Usestage.static_labels.
Mind cardinality: each label combination creates a stream in Loki.trace_idor a user ID as a label quickly degrades performance: that is the role ofstage.structured_metadata.
Multi-line stack traces #
Without care, each line of a Java stack becomes a separate log. stage.multiline groups them: a new entry starts at each line matching firstline.
loki.process "java" {
forward_to = [loki.write.local.receiver]
stage.multiline {
firstline = "^\\d{4}-\\d{2}-\\d{2}"
max_wait_time = "3s"
}
stage.regex {
expression = "^\\S+ \\S+ (?P<level>[A-Z]+) "
}
stage.labels {
values = { level = "" }
}
}
loki.write "local" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}Tested: the 3 stack lines form a single entry, with level=ERROR.
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Alloy : recevoir de l'OTLP et l'envoyer vers Tempo et Mimir✓ solved · 1 reply(ies)
Ask a question about this lesson →See all discussions about this course
Course Glossary
Grafana Alloy
An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).
Recommended Articles
Grafana Alloy components reference
A complete reference guide to all the components available in Grafana Alloy, organized by categor...
Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry
Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...
Grafana Alloy: collecting and transforming telemetry
Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...
OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines
Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...
Grafana Alloy vs Grafana Agent vs Promtail: which one to choose?
Discover how telemetry collectors have evolved in the Grafana ecosystem. A detailed comparison of...
Grafana Alloy: collecting system metrics and local logs
Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...
Grafana Alloy: The importance of Self-Monitoring
Discover why and how to configure Grafana Alloy so that it monitors itself, collecting its own lo...
The LGTM stack (Loki, Grafana, Tempo, Mimir): architecture and setup
Grafana Labs' LGTM stack: Loki for logs, Grafana for visualization, Tempo for traces, Mimir for m...
Grafana Alloy: Trace Management with Tempo
Dive into distributed trace processing. Learn how to ingest OTLP, Jaeger, or Zipkin traces with G...