Lab: Correlate the four signals of two auto-instrumented services

What you will learn in this TP

• Auto-instrument every signal
• Ensure a shared resource
• Navigate between signals

Introduction #

What you will learn in this section

• Auto-instrument every signal
• Ensure a shared resource
• Navigate between signals

Context. A two-service shop returns intermittent errors. You instrument without touching code and investigate by navigating from one signal to another.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

A cause found in a few clicks thanks to a shared resource and shared identifiers.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.

2. 2. Write the two services

   Why? The code has no OpenTelemetry line: everything will come from auto-instrumentation.

   Create front.py (port 8001) calling stock.py (port 8002); the stock service fails 15% of requests with an error log.

3. 3. Start with a shared resource

   Why? The same variables for all three signals ensure the same resource: that is the key to correlation.

   Start each service with opentelemetry-instrument, the three OTLP exporters, log capture and shared resource attributes, then generate traffic.

4. 4. Start from the metric

   Why? The investigation starts like an alert: an error metric from the HTTP conventions, the same for every service.

   In Prometheus, compute the 5xx error rate per service from the http.server.request.duration histogram (or its equivalent depending on the instrumentation version). Turn on exemplars and open a trace from a point.

5. 5. From trace to logs

   Why? The trace shows which service failed; its logs say why.

   In Tempo, search failed traces, open one, see the stock server span is 503 and use Logs for this span to find the "connection pool exhausted" message. Do the reverse path from Loki.

6. 6. Service map and span metrics

   Why? Metrics computed from spans give rate, errors and duration per service and a dependency map.

   Open Tempo's Service Graph tab in Explore: check the front to stock edge and its error rate. Note the configuration line (servicegraph connector or Tempo metrics generator) that enables it.

7. 7. Break then fix correlation

   Why? A different resource is enough to break every link: verify it to remember it.

   Restart stock with OTEL_LOGS_EXPORTER=none, regenerate traffic and see that Logs for this span returns nothing. Restore the configuration and stop everything.

Difficulty level : ●●●○○ (3/5)