Grafana Loki best practices: labels, queries, storage and costs

1. Labels: the golden rule #

Few, stable, bounded.

Choosing your labels #

Available in: ✓ OSS ✓ Enterprise ✓ Cloud Loki open source, Enterprise Logs and Grafana Cloud Logs

# Analyser la cardinalité
logcli series '{env="prod"}' --since=1h --analyze-labels
logcli labels pod --since=24h | wc -l
sum(loki_ingester_memory_streams)
# Déplacer l'information à forte cardinalité en structured metadata
stage.structured_metadata { values = { trace_id = "" } }

Common mistake: Do not exceed a few tens of thousands of active streams per tenant without sizing the ingesters accordingly.

2. Usable logs #

Structuring at the source.

Format and levels #

• Log in JSON or logfmt with level, msg, trace_id, service.
• One line = one event; avoid ungrouped multiline stack traces (stage.multiline).
• Normalize levels (debug, info, warn, error) and reduce the debug level in production.

loki.process "app" {
  forward_to = [loki.write.default.receiver]
  stage.multiline { firstline = "^\\d{4}-\\d{2}-\\d{2}"  max_wait_time = "3s" }
  stage.json      { expressions = { level = "level", trace_id = "trace_id" } }
  stage.labels    { values = { level = "" } }
  stage.structured_metadata { values = { trace_id = "" } }
  stage.drop      { expression = ".*healthcheck.*" }
}

3. Fast queries #

Efficient LogQL.

Rules #

# ✅ Sélecteur précis + filtre de ligne AVANT le parseur
{app="checkout", env="prod"} |= "error" | json | status >= 500
# ❌ Sélecteur large puis tout filtrer après parsing
{env="prod"} | json | app="checkout" | status >= 500

# Mesurer ce qu'une requête lit
logcli query '{app="checkout"} |= "error"' --since=6h --stats
# Précalculer ce qui est demandé souvent (recording rule)
- record: app:log_errors:rate5m
  expr: sum by (app) (rate({env="prod"} |= "error" [5m]))

• Reduce the time range and the number of streams before refining.
• Prefer pattern or logfmt to regexp.
• Enable the results cache and the Query Frontend for dashboards.

4. Costs, security and review #

Staying in control.

Retention and limits #

limits_config:
  retention_period: 744h
  retention_stream:
    - { selector: '{env="dev"}',   priority: 1, period: 72h }
    - { selector: '{app="audit"}', priority: 2, period: 8760h }
  ingestion_rate_mb: 10
  per_stream_rate_limit: 5MB
  max_streams_per_user: 100000
# Suivre ce qui est rejeté
sum by (reason) (rate(loki_discarded_samples_total[5m]))

Good practice: Measure the cost per application (bytes ingested) and share it with the teams: it is the best lever for reducing noise.

Reference diagram and quick check #

Visualize the principle and check it from the command line.

Reference diagram #

sum(loki_ingester_memory_streams)
logcli series '{env="prod"}' --since=1h --analyze-labels