Lab: Hand-instrument traces (spans, events, errors, links, sampling)

What you will learn in this TP

• Create spans with the SDK
• Report errors, events and links
• Query with TraceQL and sample

Introduction #

What you will learn in this section

• Create spans with the SDK
• Report errors, events and links
• Query with TraceQL and sample

Context. The order service has no auto-instrumented library for its business logic. You instrument it by hand to see an order's steps, its payment errors and its batch processing.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

Rich, queryable traces and controlled sampling.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.

2. 2. Initialise the SDK

   Why? The TracerProvider holds the resource and exporter; the BatchSpanProcessor sends in batches.

   Install the SDK and the OTLP/HTTP exporter, then write otel_setup.py configuring the resource (service.name, service.version, deployment.environment.name) and export.

3. 3. Create a business trace

   Why? Spans named per step, with SpanKind and convention-compliant attributes, make the trace readable by every tool.

   Write commande.py: a root span traiter commande and three children (stock as CLIENT with database attributes, computation, payment), an event, and a payment error one time in three with record_exception and ERROR status.

4. 4. Explore and query with TraceQL

   Why? TraceQL filters on span structure and attributes: one query finds the traces that matter.

   In Grafana, open Explore > Tempo. Run the queries below, open a failed trace and find the event and exception of the payer span.

5. 5. Link a batch job with links

   Why? A batch processing items from several traces cannot have several parents: use links.

   Write lot.py: three independent "reception" traces, then a facturer lot span linked to all three. Find the links in the span details.

6. 6. Sample

   Why? Ratio sampling reduces volume; parentbased ensures a trace is kept or dropped as a whole.

   Re-run commande.py with 20% sampling and compare the number of traces received; check no trace is partial. Explain why Collector tail sampling would be better to keep all errors.

Difficulty level : ●●●○○ (3/5)