Lab: Hand-instrument traces (spans, events, errors, links, sampling) with correction

What you will learn in this TP

• Create spans with the SDK
• Report errors, events and links
• Query with TraceQL and sample

Introduction #

What you will learn in this section

• Create spans with the SDK
• Report errors, events and links
• Query with TraceQL and sample

Context. The order service has no auto-instrumented library for its business logic. You instrument it by hand to see an order's steps, its payment errors and its batch processing.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 50 min

Rich, queryable traces and controlled sampling.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
   See the correction

   docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 -p 4040:4040 grafana/otel-lgtm
   python3 -m venv .venv && . .venv/bin/activate
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   # Grafana : http://localhost:3000 (admin / admin)

2. 2. Initialise the SDK

   Why? The TracerProvider holds the resource and exporter; the BatchSpanProcessor sends in batches.

   Install the SDK and the OTLP/HTTP exporter, then write otel_setup.py configuring the resource (service.name, service.version, deployment.environment.name) and export.
   See the correction

   pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
   cat > otel_setup.py <<'EOF'
   from opentelemetry import trace
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   
   resource = Resource.create({"service.name": "commandes", "service.version": "1.0.0",
                               "deployment.environment.name": "tp"})
   provider = TracerProvider(resource=resource)      # l'échantillonneur est lu dans OTEL_TRACES_SAMPLER
   provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(provider)
   EOF

3. 3. Create a business trace

   Why? Spans named per step, with SpanKind and convention-compliant attributes, make the trace readable by every tool.

   Write commande.py: a root span traiter commande and three children (stock as CLIENT with database attributes, computation, payment), an event, and a payment error one time in three with record_exception and ERROR status.
   See the correction

   cat > commande.py <<'EOF'
   import random, time
   from otel_setup import provider
   from opentelemetry import trace
   from opentelemetry.trace import SpanKind, Status, StatusCode
   
   tracer = trace.get_tracer("tp.commandes", "1.0.0")
   
   class PaiementRefuse(Exception):
       pass
   
   def traiter(numero):
       with tracer.start_as_current_span("traiter commande", kind=SpanKind.SERVER) as span:
           span.set_attribute("shop.order.id", numero)
           with tracer.start_as_current_span("SELECT stock", kind=SpanKind.CLIENT) as db:
               db.set_attribute("db.system.name", "postgresql")
               db.set_attribute("db.collection.name", "stock")
               db.set_attribute("db.operation.name", "SELECT")
               time.sleep(random.uniform(0.01, 0.05))
           with tracer.start_as_current_span("calculer total"):
               time.sleep(random.uniform(0.005, 0.02))
           with tracer.start_as_current_span("payer", kind=SpanKind.CLIENT) as pay:
               pay.add_event("fraude.verifiee", {"shop.fraud.score": round(random.random(), 2)})
               time.sleep(random.uniform(0.05, 0.3))
               try:
                   if numero % 3 == 0:
                       raise PaiementRefuse("carte refusée")
               except PaiementRefuse as exc:
                   pay.record_exception(exc)
                   pay.set_status(Status(StatusCode.ERROR, str(exc)))
                   span.set_status(Status(StatusCode.ERROR, "paiement refusé"))
                   span.set_attribute("error.type", "PaiementRefuse")
   
   for n in range(1, 31):
       traiter(n)
   provider.shutdown()      # vide le dernier lot
   EOF
   python commande.py

4. 4. Explore and query with TraceQL

   Why? TraceQL filters on span structure and attributes: one query finds the traces that matter.

   In Grafana, open Explore > Tempo. Run the queries below, open a failed trace and find the event and exception of the payer span.
   See the correction

   { resource.service.name = "commandes" && status = error }
   { name = "payer" && duration > 200ms }
   { span.db.system.name = "postgresql" }
   { name = "traiter commande" } >> { name = "payer" && status = error }

5. 5. Link a batch job with links

   Why? A batch processing items from several traces cannot have several parents: use links.

   Write lot.py: three independent "reception" traces, then a facturer lot span linked to all three. Find the links in the span details.
   See the correction

   cat > lot.py <<'EOF'
   from otel_setup import provider
   from opentelemetry import trace
   from opentelemetry.trace import Link
   
   tracer = trace.get_tracer("tp.lots")
   contextes = []
   for i in range(3):
       with tracer.start_as_current_span("recevoir message") as span:
           span.set_attribute("messaging.message.id", f"msg-{i}")
           contextes.append(span.get_span_context())
   with tracer.start_as_current_span("facturer lot", links=[Link(c) for c in contextes]) as lot:
       lot.set_attribute("messaging.batch.message_count", len(contextes))
   provider.shutdown()
   EOF
   python lot.py

6. 6. Sample

   Why? Ratio sampling reduces volume; parentbased ensures a trace is kept or dropped as a whole.

   Re-run commande.py with 20% sampling and compare the number of traces received; check no trace is partial. Explain why Collector tail sampling would be better to keep all errors.
   See the correction

   OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.2 python commande.py
   # Grafana : { resource.service.name = "commandes" } sur les 5 dernières minutes : ~6 nouvelles traces au lieu de 30
   # le head sampling jette aussi ~80 % des traces en erreur : le tail sampling (politique status_code) les garderait toutes

Difficulty level : ●●●○○ (3/5)