Sending traces to Tempo: OpenTelemetry instrumentation and Alloy

1. OpenTelemetry environment variables #

The same contract for all languages.

The variables to know #

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,service.version=1.4.2"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc                 # ou http/protobuf (4318)
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.25                     # 25 % des traces
export OTEL_PROPAGATORS=tracecontext,baggage

2. Auto-instrumentation #

Traces without changing the code.

Python, Java and Node.js #

# Python
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=checkout opentelemetry-instrument python app.py
OTEL_SERVICE_NAME=api opentelemetry-instrument gunicorn app:app

# Java : agent JVM
curl -LO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
java -javaagent:./opentelemetry-javaagent.jar -Dotel.service.name=orders -jar app.jar

# Node.js
npm install @opentelemetry/auto-instrumentations-node
OTEL_SERVICE_NAME=front node --require @opentelemetry/auto-instrumentations-node/register server.js

Good practice: On Kubernetes, the OpenTelemetry Operator injects the agent automatically with an annotation: instrumentation.opentelemetry.io/inject-python: "true".

3. Manual instrumentation #

Adding your own spans.

Python example #

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("checkout")

def payer(commande):
    with tracer.start_as_current_span("payer_commande") as span:
        span.set_attribute("commande.id", commande.id)
        span.set_attribute("commande.montant", commande.total)
        try:
            appeler_passerelle(commande)
            span.add_event("paiement_valide")
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise

Common mistake: Never put personal data (email, card number) in span attributes: they are stored and can be viewed by anyone with access to the traces.

4. Alloy as the entry point #

Processing before sending.

Alloy pipeline to Tempo #

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output { traces = [otelcol.processor.memory_limiter.default.input] }
}

otelcol.processor.memory_limiter "default" {
  check_interval = "1s"
  limit_percentage = 80
  output { traces = [otelcol.processor.batch.default.input] }
}

otelcol.processor.batch "default" {
  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls { insecure = true }
  }
}

# Vérifier que les spans arrivent
curl -s http://localhost:12345/metrics | grep -E 'otelcol_(receiver_accepted|exporter_sent)_spans'
docker compose logs tempo | grep -i "level=error" | tail