RousselTMLEARNING

Observability

Grafana Alloy: Log Management with Loki

Discover how to configure Grafana Alloy to read log files, journald, or network streams, process this data, and forward it securely to Grafana Loki.

1. Log sources #

Reading files, journald, and network receivers.

Where do logs come from? #

Available in: ✓ OSS ✓ Enterprise ✓ Cloud

The logs chain in Alloy
ComponentSource
loki.source.fileFiles, with rotation handling
loki.source.journalsystemd journal
loki.source.dockerDocker containers
loki.source.kubernetesPods, through the Kubernetes API
loki.source.syslogSyslog messages received over the network
loki.source.apiHTTP reception, in Loki format
otelcol.receiver.otlpOpenTelemetry logs from applications

Reading files #

local.file_match finds files, loki.source.file reads them. Target labels, here job, travel with each line; __path__ designates the file and becomes the filename label.

local.file_match "app" {
  path_targets = [{ "__path__" = "/var/log/app/*.log", "job" = "app" }]
}

loki.source.file "app" {
  targets    = local.file_match.app.targets
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Alloy remembers its position in --storage.path: at restart, it resumes where it stopped. If this folder is not kept, the whole file is re-read and Loki receives duplicates.

Nothing arrives? Check permissions first: the alloy user must be able to read the files. On Debian, add it to the adm group for /var/log.

The systemd journal #

loki.source.journal "systeme" {
  forward_to = [loki.write.local.receiver]
  labels     = { job = "journal" }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.source.journal can also transform internal journal fields with relabel_rules, for example to keep the unit name.

Checking without Loki #

To see what Alloy sees, plug loki.echo into forward_to: each line is written to Alloy's logs with its labels. The tests for this course relied on this component.

2. Processing pipelines #

Filtering, parsing (JSON, Regex), and relabeling.

Processing line by line #

loki.process applies stages, in order. They share a working area, the “extracted map”: one stage drops values there, a later one uses them.

loki.process "app" {
  forward_to = [loki.write.local.receiver]

  stage.json {
    expressions = { level = "", service = "", trace_id = "", ts = "", msg = "" }
  }

  stage.drop {
    source = "level"
    value  = "debug"
  }

  stage.timestamp {
    source = "ts"
    format = "RFC3339"
  }

  stage.labels {
    values = { level = "", service = "" }
  }

  stage.structured_metadata {
    values = { trace_id = "" }
  }

  stage.static_labels {
    values = { env = "production" }
  }

  stage.output {
    source = "msg"
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

What this pipeline produces #

Tested end to end with a real Loki, on this file:

{"ts":"2026-11-05T10:00:00Z","level":"error","service":"paiement","trace_id":"abc123","msg":"timeout banque"}
{"ts":"2026-11-05T10:00:00Z","level":"debug","service":"paiement","trace_id":"abc124","msg":"détail inutile"}
{"ts":"2026-11-05T10:00:00Z","level":"info","service":"panier","trace_id":"abc125","msg":"article ajouté"}
ligne non json
LineIn Loki
error, paiementText timeout banque, labels level=error, service=paiement, env=production, metadata trace_id=abc123
debugDiscarded: loki_process_dropped_lines_total is 1
info, panierText article ajouté, labels level=info, service=panier
ligne non jsonKept as is, with no level or service

The most useful stages #

StagePurpose
stage.json, stage.regex, stage.logfmtExtract values
stage.labelsTurn an extracted value into a label
stage.static_labelsAdd a label with a fixed value
stage.structured_metadataKeep a varied value without creating streams
stage.timestampUse the log time
stage.dropDiscard lines
stage.outputReplace the line text
stage.multilineGroup several lines
Do not confuse stage.labels and stage.static_labels. In stage.labels { values = { env = "production" } }, production is the name of an extracted value, not text. alloy validate accepts it, but we checked that the label never appears. Use stage.static_labels.
Mind cardinality: each label combination creates a stream in Loki. trace_id or a user ID as a label quickly degrades performance: that is the role of stage.structured_metadata.

Multi-line stack traces #

Without care, each line of a Java stack becomes a separate log. stage.multiline groups them: a new entry starts at each line matching firstline.

loki.process "java" {
  forward_to = [loki.write.local.receiver]

  stage.multiline {
    firstline     = "^\\d{4}-\\d{2}-\\d{2}"
    max_wait_time = "3s"
  }

  stage.regex {
    expression = "^\\S+ \\S+ (?P<level>[A-Z]+) "
  }

  stage.labels {
    values = { level = "" }
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Tested: the 3 stack lines form a single entry, with level=ERROR.

3. Forwarding to Loki #

Secure forwarding to Grafana Loki.

The loki.write component #

loki.write "central" {
  external_labels = { cluster = "paris" }

  endpoint {
    url       = "https://loki.example.com/loki/api/v1/push"
    tenant_id = "equipe-paiement"

    batch_size = "1MiB"
    batch_wait = "1s"

    max_backoff_retries = 10
    max_backoff_period  = "5m"

    basic_auth {
      username = sys.env("LOKI_USER")
      password = sys.env("LOKI_PASSWORD")
    }
  }
}
SettingPurposeDefault
batch_sizeBatch size1MiB
batch_waitMaximum wait before sending1s
min_backoff_periodFirst delay before a retry500ms
max_backoff_periodMaximum delay between retries5m
max_backoff_retriesNumber of retries; 0 to retry forever10
tenant_idTenant, in multi-tenant mode–

What happens during an outage? #

Alloy retries with a growing delay. Once retries run out, the batch is lost and loki_write_dropped_entries_total increases. For file logs, the position protects what has not been read yet; for the rest, plan for the risk.

# Lines lost after retries ran out (alert on this)
rate(loki_write_dropped_entries_total[5m]) > 0

# Lines sent, retries
rate(loki_write_sent_entries_total[5m])
rate(loki_write_batch_retries_total[5m])

Several destinations #

As for metrics, forward_to accepts several receivers: one loki.write for current storage, another for audit, with longer retention on the server side. external_labels are added when sending: keep them for what identifies the site, such as cluster.

Checking arrival #

We checked sending directly in Loki:

curl -s http://localhost:3100/loki/api/v1/labels
curl -s -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="app"}' \
  --data-urlencode "start=$(( $(date +%s)-600 ))000000000"
Best practice: keep few labels, with few values. job, env, level and service cover most needs; the rest is searched in the text at query time.

2026 update: what has changed #

Recent changes worth knowing (checked in November 2026).

Migrating from Promtail #

Promtail reached end of life on March 2, 2026: no more fixes or support. Alloy provides a converter, but a few points need a manual check:

  • The metrics exposed by Alloy differ from Promtail's: adapt alerts and dashboards.
  • The positions file is stored elsewhere: plan how reading resumes so old logs are not sent again.
  • For a gradual transition, Alloy can temporarily read a Promtail configuration (--config.format=promtail).
alloy convert --source-format=promtail --report=report.txt promtail.yaml -o config.alloy
alloy run --config.format=promtail promtail.yaml      # temporary transition
Good to know: Grafana Alloy ships a minor release roughly every three weeks; 1.21 (October 2026) is the latest at the time of this update. Read the release notes before each upgrade: some releases bring a new Prometheus or OpenTelemetry Collector branch, with flagged breaking changes.

Related articles

Alloy and the Grafana Labs ecosystem: which component for which tool?

The map of Alloy integrations with Faro, Beyla, k6, Loki, Mimir, Tempo, Pyroscope, Database Observability, Fl…

Faro and Alloy: receiving front-end RUM with faro.receiver

Configure faro.receiver: CORS, API key, rate limiting, source maps, then send browser errors, Web Vitals and …

Beyla and Alloy: zero-code RED metrics and traces with beyla.ebpf

Enable Beyla eBPF instrumentation in Alloy: service discovery, RED metrics to Mimir, traces to Tempo, require…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!