1. Anatomy of a trace #
Trace, span, context.
Trace and spans #
A trace represents the full journey of a request. It is made of spans: named operations, with a start, an end, a parent, attributes, events and a status. All spans share the same trace_id (16 bytes); each has its own span_id (8 bytes).
| Field | Role |
|---|---|
| name | operation, low cardinality: GET /users/{id} |
| trace_id, span_id, parent_span_id | tree structure |
| start / end | duration |
| kind | role of the span in the exchange |
| attributes | context (semconv) |
| events | timestamped facts during the span |
| links | relationships with other traces |
| status | Unset, Ok or Error |
SpanKind #
| Kind | When | Example |
|---|---|---|
| SERVER | receiving a synchronous request | HTTP handler, gRPC method |
| CLIENT | outgoing synchronous call | HTTP request, SQL query |
| PRODUCER | sending an asynchronous message | Kafka, RabbitMQ publication |
| CONSUMER | processing a received message | Kafka consumer |
| INTERNAL | internal operation | computation, business step |
2. Manual instrumentation #
Attributes, status, events, links.
Creating spans #
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode, Link
tracer = trace.get_tracer("shop.checkout", "1.4.0")
def payer(commande):
with tracer.start_as_current_span("payer commande", kind=SpanKind.INTERNAL) as span:
span.set_attribute("shop.order.id", commande.id) # attribut métier
span.set_attribute("shop.order.amount", commande.total)
span.add_event("fraude.verifiee", {"shop.fraud.score": 0.12})
try:
passerelle.debiter(commande)
except PaiementRefuse as exc:
span.record_exception(exc) # événement "exception"
span.set_status(Status(StatusCode.ERROR, "paiement refusé"))
raiseStatus and errors #
The status is Unset by default. Set it to Error when the operation failed; only set Ok to force success despite a rule saying otherwise. For HTTP, the conventions say: on the server side, only 5xx responses are errors; on the client side, both 4xx and 5xx are.
Common mistake: Do not use attributes to store large or personal data (request bodies, emails): cost, cardinality and GDPR.
Links #
A link connects a span to one or more other contexts without a parent-child relationship: a batch job aggregating 100 messages from 100 different traces, or an asynchronous task triggered by a request. tracer.start_as_current_span("lot", links=[Link(ctx1), Link(ctx2)]).
3. The span pipeline in the SDK #
Processors, exporters, limits.
Span processors #
| Component | Role |
|---|---|
| SimpleSpanProcessor | exports each span when it ends: only for debugging |
| BatchSpanProcessor | groups and exports in batches: to use in production |
| Limits | OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, OTEL_SPAN_EVENT_COUNT_LIMIT protect against size explosion |
Clean shutdown #
Good practice: Callprovider.shutdown()(orforce_flush()) when a short-lived process stops (job, serverless function): otherwise the last batch of spans is lost.
4. Sampling #
Keeping the right traces without paying for all of them.
Head and tail #
| Sampler (SDK) | Behavior |
|---|---|
always_on / always_off | all or nothing |
traceidratio | keeps a ratio, computed on the trace_id (same decision everywhere) |
parentbased_traceidratio | follows the parent's decision; ratio for traces starting here (recommended) |
processors:
tail_sampling:
decision_wait: 10s
policies:
- { name: erreurs, type: status_code, status_code: { status_codes: [ERROR] } }
- { name: lentes, type: latency, latency: { threshold_ms: 1000 } }
- { name: reste, type: probabilistic, probabilistic: { sampling_percentage: 5 } }Common mistake: Tail sampling requires all spans of a trace to reach the same Collector: put a Collector with the loadbalancing exporter routed by TraceID in front.Querying traces #
{ resource.service.name = "checkout" && status = error }
{ span.http.route = "/orders" && duration > 500ms }
{ span.db.system.name = "postgresql" } | count() > 20 # traces qui font plus de 20 requêtes SQL (N+1)
{ resource.service.name = "frontend" } >> { resource.service.name = "payment" && status = error }My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Quels attributs utiliser pour nommer mes spans HTTP (semconv) ?✓ solved · 1 reply(ies)
- OpenTelemetry Collector : mes traces arrivent mais pas mes métriques✓ solved · 2 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
Grafana Tempo: the distributed tracing backend at scale
Understand distributed tracing and Grafana Tempo: traces, spans, TraceID, architecture (distribut...
Distributed tracing: context propagation (W3C Trace Context and baggage)
Understand and implement OpenTelemetry context propagation: the traceparent header, tracestate, b...
Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces
Understand OpenTelemetry's new profile signal: continuous profiling, flame graphs, data model and...
TraceQL: the trace query language
Master TraceQL: span selectors, resource and span attributes, operators, duration comparisons, st...
Correlating traces, metrics, logs and profiles with OpenTelemetry
Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...
Distributed tracing best practices with Jaeger
Checklist for useful tracing: OpenTelemetry instrumentation, span naming, attributes, propagation...
Linking traces and profiles: span profiles with Tempo and Pyroscope
Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...
Traefik: distributed tracing with OpenTelemetry
Enable Traefik v3 tracing with OpenTelemetry (OTLP), propagate the W3C context to applications, s...
OpenTelemetry (OTel) best practices: the guide to modern observability
Master OpenTelemetry with our expert advice: implementing the Collector, following semantic conve...