Processing Pipelines

2. Processing pipelines #

Filtering, parsing (JSON, Regex), and relabeling.

Enrich and filter logs #

Raw logs often lack context and consume too much storage. The loki.process component allows you to define sophisticated processing pipelines. You can embed stages to extract fields via JSON or Regex parsing, use these fields to create new dynamic labels, filter (drop) irrelevant lines (like HTTP healthchecks), or even rewrite the log content itself.

Best Practice: Drastically limit the number of labels extracted to Loki. Only extract global and static fields as labels (e.g., env, app, level). For other fields (dynamic business values), leave them in the log line and use LogQL filters (e.g., | json) at query time.

Common Mistake: Promoting a highly variable field (like an IP address, a trace_id, or a transaction uuid) as a label. This mistake will crush the performance of your Loki instance.

loki.process "filter_logs" {
  forward_to = [loki.write.default.receiver]

  // JSON Parsing
  stage.json {
    expressions = { level = "log_level", app = "app_name" }
  }
  // Secure label creation
  stage.labels {
    values = { level = null, app = null }
  }
  // Filter out useless logs
  stage.drop {
    source = "level"
    value  = "debug"
  }
}