Traefik: distributed tracing with OpenTelemetry

1. Why trace the proxy #

The first link of the request.

Principle #

Traefik sees every request first. By creating the root span and passing the traceparent header (W3C Trace Context), it makes it possible to follow a request from the entry point to the database and to distinguish time spent in the proxy from time spent in the applications.

2. Configuration (Traefik v3) #

Native OpenTelemetry.

Static configuration #

tracing:
  serviceName: traefik
  sampleRate: 0.2                 # 20 % des requêtes
  addInternals: false             # ne pas tracer le tableau de bord ni /ping
  otlp:
    grpc:
      endpoint: otel-collector:4317
      insecure: true

Change in v3: Traefik v3 no longer exports directly to Jaeger, Zipkin or Datadog: it emits OTLP, and the OpenTelemetry Collector routes to the backend of your choice.

Checking #

curl -s -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' https://app.exemple.fr/ -o /dev/null
# puis rechercher la trace 4bf92f3577b34da6a3ce929d0e0e4736 dans Jaeger ou Tempo

3. Reading and using traces #

What Traefik spans show.

Analysis #

• The Traefik span contains the router, the service, the status and the backend address.
• A large gap between the duration of the Traefik span and that of the application span reveals waiting on the proxy or network side (connections, TLS, queue).
• The TraceId also appears in access logs: you can go from the log to the trace.

Good practice: Head-sample in Traefik at a reasonable rate and keep 100% of errors with the Collector's tail sampling.