Correlating traces, logs, metrics and profiles in Grafana

1. Why correlate? #

Reducing diagnosis time.

The typical path #

Available in: ✓ OSS ✓ Enterprise ✓ Cloud

An alert signals a high error rate. You open the metrics graph, click an exemplar to see a slow trace, then the faulty span to display the logs of the same pod at the same moment, and finally the CPU profile that explains the bottleneck. All this requires configured links and consistent labels.

2. Provisioning the Tempo data source #

Links to the other signals.

datasource.yml #

apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: "-5m"
        spanEndTimeShift: "5m"
        filterByTraceID: true
        filterBySpanID: false
        tags:
          - { key: service.name, value: app }
          - { key: k8s.namespace.name, value: namespace }
      tracesToMetrics:
        datasourceUid: prometheus
        tags: [{ key: service.name, value: service }]
        queries:
          - name: Taux de requêtes
            query: sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))
      tracesToProfiles:
        datasourceUid: pyroscope
        tags: [{ key: service.name, value: service_name }]
        profileTypeId: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"
      serviceMap: { datasourceUid: prometheus }
      nodeGraph: { enabled: true }

Good to know: The left-hand key (service.name) is the span attribute; the value (app) is the Loki label used to filter logs: adapt it to your labels.

3. From log to trace: derived fields #

Clicking a TraceID in a log.

Loki data source #

- name: Loki
  type: loki
  uid: loki
  url: http://loki:3100
  jsonData:
    derivedFields:
      - name: TraceID
        matcherRegex: '"trace_id":"(\\w+)"'
        url: '$${__value.raw}'
        datasourceUid: tempo
        urlDisplayLabel: Voir la trace

# Les logs doivent contenir le TraceID : exemple avec un logger Python
import logging
from opentelemetry import trace
ctx = trace.get_current_span().get_span_context()
logging.info("paiement refusé", extra={"trace_id": format(ctx.trace_id, "032x")})

Good practice: With OpenTelemetry auto-instrumentation (OTEL_PYTHON_LOG_CORRELATION=true), the TraceID and SpanID are added to logs automatically.

4. Checking consistency #

Labels must line up.

Checklist #

• The same service.name (traces) ↔ app/service (logs, metrics).
• Exemplars enabled on the Prometheus side (--enable-feature=exemplar-storage) and on the application side.
• Synchronized clocks (NTP): a time offset skews the time window of the links.
• The TraceID written in the logs as a 32-character hexadecimal string.

# Vérifier la présence du TraceID dans les logs
logcli query '{app="checkout"} |= "trace_id"' --since=10m --limit=3
# Vérifier les exemplars exposés par l'application
curl -s -H 'Accept: application/openmetrics-text' http://app:8080/metrics | grep -m3 'trace_id'