RousselTMLEARNING

TP · OpenTelemetry: From Discovery to Expertise

Lab: Correlate the four signals of two auto-instrumented services

What you will learn in this TP
  • Auto-instrument every signal
  • Ensure a shared resource
  • Navigate between signals

Introduction #

What you will learn in this section
  • Auto-instrument every signal
  • Ensure a shared resource
  • Navigate between signals

Context. A two-service shop returns intermittent errors. You instrument without touching code and investigate by navigating from one signal to another.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

A cause found in a few clicks thanks to a shared resource and shared identifiers.

Hands-on #

Guided steps, with their solution.

  1. 1. Start the observability stack

    Why? A single image provides an OTLP Collector and every backend: focus on the signal.

    Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
  2. 2. Write the two services

    Why? The code has no OpenTelemetry line: everything will come from auto-instrumentation.

    Create front.py (port 8001) calling stock.py (port 8002); the stock service fails 15% of requests with an error log.
  3. 3. Start with a shared resource

    Why? The same variables for all three signals ensure the same resource: that is the key to correlation.

    Start each service with opentelemetry-instrument, the three OTLP exporters, log capture and shared resource attributes, then generate traffic.
  4. 4. Start from the metric

    Why? The investigation starts like an alert: an error metric from the HTTP conventions, the same for every service.

    In Prometheus, compute the 5xx error rate per service from the http.server.request.duration histogram (or its equivalent depending on the instrumentation version). Turn on exemplars and open a trace from a point.
  5. 5. From trace to logs

    Why? The trace shows which service failed; its logs say why.

    In Tempo, search failed traces, open one, see the stock server span is 503 and use Logs for this span to find the "connection pool exhausted" message. Do the reverse path from Loki.
  6. 6. Service map and span metrics

    Why? Metrics computed from spans give rate, errors and duration per service and a dependency map.

    Open Tempo's Service Graph tab in Explore: check the front to stock edge and its error rate. Note the configuration line (servicegraph connector or Tempo metrics generator) that enables it.
  7. 7. Break then fix correlation

    Why? A different resource is enough to break every link: verify it to remember it.

    Restart stock with OTEL_LOGS_EXPORTER=none, regenerate traffic and see that Logs for this span returns nothing. Restore the configuration and stop everything.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 50 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

OpenTelemetry metrics: instruments, temporality, histograms and exemplars

Master the metric signal: MeterProvider, synchronous and asynchronous instruments (Counter, UpDow...

Logs with OpenTelemetry: data model, appenders, correlation and collection

Master the log signal: the LogRecord data model, the Logs Bridge API and appenders (Logback, Log4...

Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack

A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...

Datadog APM and OpenTelemetry: traces, services and correlation with logs

Instrument applications for Datadog APM (dd-trace, single step instrumentation, OpenTelemetry), u...

Core OpenTelemetry concepts: signals, resources, context and maturity

The essential OpenTelemetry vocabulary: the four signals, API and SDK, resource and instrumentati...

OpenTelemetry: automatic vs manual instrumentation

Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...

Observability on AWS: CloudWatch, X-Ray, Application Signals, ADOT, Managed Prometheus and Grafana

A complete overview of observability on AWS: CloudWatch components (metrics, logs, alarms, Applic...

Correlating traces, logs, metrics and profiles in Grafana

Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!