Lab: Correlate the four signals of two auto-instrumented services with correction

What you will learn in this TP

• Auto-instrument every signal
• Ensure a shared resource
• Navigate between signals

Introduction #

What you will learn in this section

• Auto-instrument every signal
• Ensure a shared resource
• Navigate between signals

Context. A two-service shop returns intermittent errors. You instrument without touching code and investigate by navigating from one signal to another.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

A cause found in a few clicks thanks to a shared resource and shared identifiers.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
   See the correction

   docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 -p 4040:4040 grafana/otel-lgtm
   python3 -m venv .venv && . .venv/bin/activate
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   # Grafana : http://localhost:3000 (admin / admin)

2. 2. Write the two services

   Why? The code has no OpenTelemetry line: everything will come from auto-instrumentation.

   Create front.py (port 8001) calling stock.py (port 8002); the stock service fails 15% of requests with an error log.
   See the correction

   pip install flask requests opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install          # installe les instrumentations Flask, requests, logging…
   cat > stock.py <<'EOF'
   import logging, random, time
   from flask import Flask
   app = Flask(__name__)
   log = logging.getLogger("stock")
   @app.get("/stock/<sku>")
   def stock(sku):
       time.sleep(random.uniform(0.01, 0.08))
       if random.random() < 0.15:
           log.error("pool de connexions épuisé pour %s", sku)
           return {"error": "indisponible"}, 503
       return {"sku": sku, "qty": random.randint(0, 40)}
   app.run(port=8002)
   EOF
   cat > front.py <<'EOF'
   import logging, random, requests
   from flask import Flask
   app = Flask(__name__)
   log = logging.getLogger("front")
   @app.get("/produit")
   def produit():
       sku = f"SKU-{random.randint(1, 50)}"
       r = requests.get(f"http://localhost:8002/stock/{sku}", timeout=2)
       if r.status_code != 200:
           log.warning("stock indisponible pour %s", sku)
           return {"error": "réessayez"}, 502
       return r.json()
   app.run(port=8001)
   EOF

3. 3. Start with a shared resource

   Why? The same variables for all three signals ensure the same resource: that is the key to correlation.

   Start each service with opentelemetry-instrument, the three OTLP exporters, log capture and shared resource attributes, then generate traffic.
   See the correction

   export OTEL_TRACES_EXPORTER=otlp OTEL_METRICS_EXPORTER=otlp OTEL_LOGS_EXPORTER=otlp
   export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true OTEL_METRIC_EXPORT_INTERVAL=5000
   export OTEL_RESOURCE_ATTRIBUTES=service.namespace=boutique,service.version=1.3.0,deployment.environment.name=tp
   OTEL_SERVICE_NAME=stock opentelemetry-instrument python stock.py &
   OTEL_SERVICE_NAME=front opentelemetry-instrument python front.py &
   sleep 3 ; for i in $(seq 1 400); do curl -s -o /dev/null localhost:8001/produit; done

4. 4. Start from the metric

   Why? The investigation starts like an alert: an error metric from the HTTP conventions, the same for every service.

   In Prometheus, compute the 5xx error rate per service from the http.server.request.duration histogram (or its equivalent depending on the instrumentation version). Turn on exemplars and open a trace from a point.
   See the correction

   sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[1m]))
     / sum by (job) (rate(http_server_request_duration_seconds_count[1m]))
   # anciennes conventions : http_server_duration_milliseconds_count{http_status_code=~"5.."}

5. 5. From trace to logs

   Why? The trace shows which service failed; its logs say why.

   In Tempo, search failed traces, open one, see the stock server span is 503 and use Logs for this span to find the "connection pool exhausted" message. Do the reverse path from Loki.
   See the correction

   { resource.service.namespace = "boutique" && status = error }
   { resource.service.name = "front" } >> { resource.service.name = "stock" && span.http.response.status_code = 503 }
   # Loki : {service_name="stock"} | severity_text="ERROR"  -> champ trace_id -> lien Tempo

6. 6. Service map and span metrics

   Why? Metrics computed from spans give rate, errors and duration per service and a dependency map.

   Open Tempo's Service Graph tab in Explore: check the front to stock edge and its error rate. Note the configuration line (servicegraph connector or Tempo metrics generator) that enables it.
   See the correction

   traces_service_graph_request_failed_total{client="front", server="stock"}
   traces_spanmetrics_calls_total{service="stock", status_code="STATUS_CODE_ERROR"}
   # selon la version de l'image, ces métriques viennent du generator de Tempo ou des connecteurs du Collector

7. 7. Break then fix correlation

   Why? A different resource is enough to break every link: verify it to remember it.

   Restart stock with OTEL_LOGS_EXPORTER=none, regenerate traffic and see that Logs for this span returns nothing. Restore the configuration and stop everything.
   See the correction

   pkill -f "python stock.py"
   OTEL_LOGS_EXPORTER=none OTEL_SERVICE_NAME=stock opentelemetry-instrument python stock.py &
   for i in $(seq 1 100); do curl -s -o /dev/null localhost:8001/produit; done
   pkill -f "opentelemetry-instrument python" ; pkill -f "python front.py" ; pkill -f "python stock.py"

Difficulty level : ●●●○○ (3/5)