What you will learn in this TP
- Auto-instrument every signal
- Ensure a shared resource
- Navigate between signals
Introduction #
What you will learn in this section
- Auto-instrument every signal
- Ensure a shared resource
- Navigate between signals
Context. A two-service shop returns intermittent errors. You instrument without touching code and investigate by navigating from one signal to another.
Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container) · Estimated time: 50 min
Hands-on #
Guided steps, with their solution.
1. Start the observability stack
Why? A single image provides an OTLP Collector and every backend: focus on the signal.
Startgrafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.2. Write the two services
Why? The code has no OpenTelemetry line: everything will come from auto-instrumentation.
Createfront.py(port 8001) callingstock.py(port 8002); the stock service fails 15% of requests with an error log.3. Start with a shared resource
Why? The same variables for all three signals ensure the same resource: that is the key to correlation.
Start each service withopentelemetry-instrument, the three OTLP exporters, log capture and shared resource attributes, then generate traffic.4. Start from the metric
Why? The investigation starts like an alert: an error metric from the HTTP conventions, the same for every service.
In Prometheus, compute the 5xx error rate per service from thehttp.server.request.durationhistogram (or its equivalent depending on the instrumentation version). Turn on exemplars and open a trace from a point.5. From trace to logs
Why? The trace shows which service failed; its logs say why.
In Tempo, search failed traces, open one, see thestockserver span is 503 and use Logs for this span to find the "connection pool exhausted" message. Do the reverse path from Loki.6. Service map and span metrics
Why? Metrics computed from spans give rate, errors and duration per service and a dependency map.
Open Tempo's Service Graph tab in Explore: check thefronttostockedge and its error rate. Note the configuration line (servicegraphconnector or Tempo metrics generator) that enables it.7. Break then fix correlation
Why? A different resource is enough to break every link: verify it to remember it.
RestartstockwithOTEL_LOGS_EXPORTER=none, regenerate traffic and see that Logs for this span returns nothing. Restore the configuration and stop everything.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Quels attributs utiliser pour nommer mes spans HTTP (semconv) ?✓ solved · 1 reply(ies)
- OpenTelemetry Collector : mes traces arrivent mais pas mes métriques✓ solved · 2 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
Correlating traces, metrics, logs and profiles with OpenTelemetry
Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...
OpenTelemetry metrics: instruments, temporality, histograms and exemplars
Master the metric signal: MeterProvider, synchronous and asynchronous instruments (Counter, UpDow...
Logs with OpenTelemetry: data model, appenders, correlation and collection
Master the log signal: the LogRecord data model, the Logs Bridge API and appenders (Logback, Log4...
Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack
A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...
Datadog APM and OpenTelemetry: traces, services and correlation with logs
Instrument applications for Datadog APM (dd-trace, single step instrumentation, OpenTelemetry), u...
Core OpenTelemetry concepts: signals, resources, context and maturity
The essential OpenTelemetry vocabulary: the four signals, API and SDK, resource and instrumentati...
OpenTelemetry: automatic vs manual instrumentation
Understand the differences between automatic and manual instrumentation with OpenTelemetry, their...
Observability on AWS: CloudWatch, X-Ray, Application Signals, ADOT, Managed Prometheus and Grafana
A complete overview of observability on AWS: CloudWatch components (metrics, logs, alarms, Applic...
Correlating traces, logs, metrics and profiles in Grafana
Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...