RousselTMLEARNING

Article · OpenTelemetry: From Discovery to Expertise

Correlating traces, metrics, logs and profiles

1. Why correlate #

Diagnosis time is the real cost of an incident.

Without and with correlation #

Without correlation, the on-call engineer sees an error spike on a dashboard, searches logs around the same time, guesses which service is at fault, then looks for a trace « that looks similar ». With correlated signals, each step is a click, on data that concerns the same request.

Navigation between signals
Each arrow corresponds to an identifier shared by the signals.

2. Correlation mechanisms #

What needs to be set up.

Overview #

LinkMechanismTo check
All signals of a serviceidentical resource (service.name, service.version, deployment.environment.name, k8s.pod.name)same SDK or same variables for all signals; k8sattributes in the Collector
Log → tracetrace_id and span_id in every logOTel appender or added to the JSON format
Metric → traceexemplars with trace_idexemplars enabled in the SDK and stored by the backend (Prometheus --enable-feature=exemplar-storage)
Trace → metricsspan metrics and service graph computed from spansspanmetrics, servicegraph connectors or Tempo's metrics generator
Trace → profilesamples tagged with the span_idprofiling SDK with a span processor
Business contextpropagated baggage (tenant.id, shop.customer.tier)processor that copies baggage into attributes

A shared resource #

# Les mêmes variables pour tous les signaux d'un service
export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=service.namespace=shop,service.version=1.4.0,deployment.environment.name=production
export OTEL_TRACES_EXPORTER=otlp OTEL_METRICS_EXPORTER=otlp OTEL_LOGS_EXPORTER=otlp
Common mistake: If traces say checkout and logs say checkout-api, no automatic navigation works. The service name is the key to all correlation.

3. Configuring Grafana #

Links between data sources.

Provisioning data sources #

apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        spanStartTimeShift: -5m
        spanEndTimeShift: 5m
      tracesToMetrics: { datasourceUid: prometheus }
      tracesToProfiles: { datasourceUid: pyroscope, profileTypeId: "process_cpu:cpu:nanoseconds:cpu:nanoseconds" }
      serviceMap: { datasourceUid: prometheus }
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - { name: TraceID, matcherRegex: '"trace_id":"(\w+)"', url: "$${__value.raw}", datasourceUid: tempo }
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations: [{ name: trace_id, datasourceUid: tempo }]

The typical journey #

Diagnosis journey
Without correlation, each step is a manual search by timestamp.
  1. The burn rate alert fires on the latency SLO.
  2. The p99 latency panel shows exemplars: click the highest one.
  3. The trace opens: the payment span lasts 2 s and ends in error.
  4. « Logs for this span » shows the connection pool exhausted log.
  5. « Profiles for this span » shows the time spent waiting for the pool.
  6. The version (service.version) shows the problem appeared with the last deployment.

4. Baggage and business context #

Correlating with the business too.

Using baggage #

from opentelemetry import baggage, context
ctx = baggage.set_baggage("shop.customer.tier", "premium")
token = context.attach(ctx)        # propagé aux services appelés (en-tête baggage)
try:
    appeler_paiement()
finally:
    context.detach(token)

In the SDKs and the Collector, a span processor can copy certain baggage keys into span attributes to filter « all slow traces from premium customers ».

Common mistake: Baggage travels in plain text in HTTP headers, including to third-party services: never put sensitive data in it and filter it when leaving the system.

📄 Correlating traces, metrics, logs and profiles with OpenTelemetry →

My progress

  • ⏱ Estimated time : ≈ 9 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

Correlating traces, logs, metrics and profiles in Grafana

Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...

Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces

Understand OpenTelemetry's new profile signal: continuous profiling, flame graphs, data model and...

Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack

A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...

Logs with OpenTelemetry: data model, appenders, correlation and collection

Master the log signal: the LogRecord data model, the Logs Bridge API and appenders (Logback, Log4...

Tempo: generating metrics from traces (service graph, span metrics, exemplars)

Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...

Linking traces and profiles: span profiles with Tempo and Pyroscope

Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...

OpenTelemetry metrics: instruments, temporality, histograms and exemplars

Master the metric signal: MeterProvider, synchronous and asynchronous instruments (Counter, UpDow...

OpenTelemetry: automatic vs manual instrumentation

Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...

Datadog APM and OpenTelemetry: traces, services and correlation with logs

Instrument applications for Datadog APM (dd-trace, single step instrumentation, OpenTelemetry), u...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!