RousselTMLEARNING

Article · Alloy: From Discovery to Expertise

Processing Pipelines

2. Processing pipelines #

Filtering, parsing (JSON, Regex), and relabeling.

Processing line by line #

loki.process applies stages, in order. They share a working area, the “extracted map”: one stage drops values there, a later one uses them.

loki.process "app" {
  forward_to = [loki.write.local.receiver]

  stage.json {
    expressions = { level = "", service = "", trace_id = "", ts = "", msg = "" }
  }

  stage.drop {
    source = "level"
    value  = "debug"
  }

  stage.timestamp {
    source = "ts"
    format = "RFC3339"
  }

  stage.labels {
    values = { level = "", service = "" }
  }

  stage.structured_metadata {
    values = { trace_id = "" }
  }

  stage.static_labels {
    values = { env = "production" }
  }

  stage.output {
    source = "msg"
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

What this pipeline produces #

Tested end to end with a real Loki, on this file:

{"ts":"2026-11-05T10:00:00Z","level":"error","service":"paiement","trace_id":"abc123","msg":"timeout banque"}
{"ts":"2026-11-05T10:00:00Z","level":"debug","service":"paiement","trace_id":"abc124","msg":"détail inutile"}
{"ts":"2026-11-05T10:00:00Z","level":"info","service":"panier","trace_id":"abc125","msg":"article ajouté"}
ligne non json
LineIn Loki
error, paiementText timeout banque, labels level=error, service=paiement, env=production, metadata trace_id=abc123
debugDiscarded: loki_process_dropped_lines_total is 1
info, panierText article ajouté, labels level=info, service=panier
ligne non jsonKept as is, with no level or service

The most useful stages #

StagePurpose
stage.json, stage.regex, stage.logfmtExtract values
stage.labelsTurn an extracted value into a label
stage.static_labelsAdd a label with a fixed value
stage.structured_metadataKeep a varied value without creating streams
stage.timestampUse the log time
stage.dropDiscard lines
stage.outputReplace the line text
stage.multilineGroup several lines
Do not confuse stage.labels and stage.static_labels. In stage.labels { values = { env = "production" } }, production is the name of an extracted value, not text. alloy validate accepts it, but we checked that the label never appears. Use stage.static_labels.
Mind cardinality: each label combination creates a stream in Loki. trace_id or a user ID as a label quickly degrades performance: that is the role of stage.structured_metadata.

Multi-line stack traces #

Without care, each line of a Java stack becomes a separate log. stage.multiline groups them: a new entry starts at each line matching firstline.

loki.process "java" {
  forward_to = [loki.write.local.receiver]

  stage.multiline {
    firstline     = "^\\d{4}-\\d{2}-\\d{2}"
    max_wait_time = "3s"
  }

  stage.regex {
    expression = "^\\S+ \\S+ (?P<level>[A-Z]+) "
  }

  stage.labels {
    values = { level = "" }
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Tested: the 3 stack lines form a single entry, with level=ERROR.

📄 Grafana Alloy: Log Management with Loki →

My progress

  • ⏱ Estimated time : ≈ 9 min

Sign in to track your progress and take notes. Sign in

⭐ Rate this lesson

5.0 initial rating from the teaching team

Sign in to rate this lesson →

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Course Glossary

Grafana Alloy

An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).

Recommended Articles

Grafana Alloy components reference

A complete reference guide to all the components available in Grafana Alloy, organized by categor...

Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry

Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...

Grafana Alloy: collecting and transforming telemetry

Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...

OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines

Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...

Grafana Alloy vs Grafana Agent vs Promtail: which one to choose?

Discover how telemetry collectors have evolved in the Grafana ecosystem. A detailed comparison of...

Grafana Alloy: collecting system metrics and local logs

Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...

Grafana Alloy: The importance of Self-Monitoring

Discover why and how to configure Grafana Alloy so that it monitors itself, collecting its own lo...

The LGTM stack (Loki, Grafana, Tempo, Mimir): architecture and setup

Grafana Labs' LGTM stack: Loki for logs, Grafana for visualization, Tempo for traces, Mimir for m...

Grafana Alloy: Trace Management with Tempo

Dive into distributed trace processing. Learn how to ingest OTLP, Jaeger, or Zipkin traces with G...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!