RousselTMLEARNING

Article · OpenTelemetry: From Discovery to Expertise

Traces in depth

1. Anatomy of a trace #

Trace, span, context.

Trace and spans #

Distributed trace
The waterfall view immediately shows that payment takes most of the time and ends in error.

A trace represents the full journey of a request. It is made of spans: named operations, with a start, an end, a parent, attributes, events and a status. All spans share the same trace_id (16 bytes); each has its own span_id (8 bytes).

FieldRole
nameoperation, low cardinality: GET /users/{id}
trace_id, span_id, parent_span_idtree structure
start / endduration
kindrole of the span in the exchange
attributescontext (semconv)
eventstimestamped facts during the span
linksrelationships with other traces
statusUnset, Ok or Error

SpanKind #

KindWhenExample
SERVERreceiving a synchronous requestHTTP handler, gRPC method
CLIENToutgoing synchronous callHTTP request, SQL query
PRODUCERsending an asynchronous messageKafka, RabbitMQ publication
CONSUMERprocessing a received messageKafka consumer
INTERNALinternal operationcomputation, business step

2. Manual instrumentation #

Attributes, status, events, links.

Creating spans #

from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode, Link

tracer = trace.get_tracer("shop.checkout", "1.4.0")

def payer(commande):
    with tracer.start_as_current_span("payer commande", kind=SpanKind.INTERNAL) as span:
        span.set_attribute("shop.order.id", commande.id)          # attribut métier
        span.set_attribute("shop.order.amount", commande.total)
        span.add_event("fraude.verifiee", {"shop.fraud.score": 0.12})
        try:
            passerelle.debiter(commande)
        except PaiementRefuse as exc:
            span.record_exception(exc)                            # événement "exception"
            span.set_status(Status(StatusCode.ERROR, "paiement refusé"))
            raise

Status and errors #

The status is Unset by default. Set it to Error when the operation failed; only set Ok to force success despite a rule saying otherwise. For HTTP, the conventions say: on the server side, only 5xx responses are errors; on the client side, both 4xx and 5xx are.

Common mistake: Do not use attributes to store large or personal data (request bodies, emails): cost, cardinality and GDPR.

Links #

A link connects a span to one or more other contexts without a parent-child relationship: a batch job aggregating 100 messages from 100 different traces, or an asynchronous task triggered by a request. tracer.start_as_current_span("lot", links=[Link(ctx1), Link(ctx2)]).

3. The span pipeline in the SDK #

Processors, exporters, limits.

Span processors #

ComponentRole
SimpleSpanProcessorexports each span when it ends: only for debugging
BatchSpanProcessorgroups and exports in batches: to use in production
LimitsOTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, OTEL_SPAN_EVENT_COUNT_LIMIT protect against size explosion

Clean shutdown #

Good practice: Call provider.shutdown() (or force_flush()) when a short-lived process stops (job, serverless function): otherwise the last batch of spans is lost.

4. Sampling #

Keeping the right traces without paying for all of them.

Head and tail #

Head or tail sampling
A common approach: 100% at the head, tail sampling at the gateway Collector.
Sampler (SDK)Behavior
always_on / always_offall or nothing
traceidratiokeeps a ratio, computed on the trace_id (same decision everywhere)
parentbased_traceidratiofollows the parent's decision; ratio for traces starting here (recommended)
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - { name: erreurs, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: lentes,  type: latency, latency: { threshold_ms: 1000 } }
      - { name: reste,   type: probabilistic, probabilistic: { sampling_percentage: 5 } }
Common mistake: Tail sampling requires all spans of a trace to reach the same Collector: put a Collector with the loadbalancing exporter routed by TraceID in front.

Querying traces #

{ resource.service.name = "checkout" && status = error }
{ span.http.route = "/orders" && duration > 500ms }
{ span.db.system.name = "postgresql" } | count() > 20          # traces qui font plus de 20 requêtes SQL (N+1)
{ resource.service.name = "frontend" } >> { resource.service.name = "payment" && status = error }

📄 OpenTelemetry traces in depth: spans, status, events, links and sampling →

My progress

  • ⏱ Estimated time : ≈ 7 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

Grafana Tempo: the distributed tracing backend at scale

Understand distributed tracing and Grafana Tempo: traces, spans, TraceID, architecture (distribut...

Distributed tracing: context propagation (W3C Trace Context and baggage)

Understand and implement OpenTelemetry context propagation: the traceparent header, tracestate, b...

Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces

Understand OpenTelemetry's new profile signal: continuous profiling, flame graphs, data model and...

TraceQL: the trace query language

Master TraceQL: span selectors, resource and span attributes, operators, duration comparisons, st...

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

Distributed tracing best practices with Jaeger

Checklist for useful tracing: OpenTelemetry instrumentation, span naming, attributes, propagation...

Linking traces and profiles: span profiles with Tempo and Pyroscope

Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...

Traefik: distributed tracing with OpenTelemetry

Enable Traefik v3 tracing with OpenTelemetry (OTLP), propagate the W3C context to applications, s...

OpenTelemetry (OTel) best practices: the guide to modern observability

Master OpenTelemetry with our expert advice: implementing the Collector, following semantic conve...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!