1. Trace Ingestion
Ingestion via OTLP or third-party protocols (Jaeger, Zipkin).
Receiving telemetry
Grafana Alloy integrates the capabilities of the OpenTelemetry Collector. By using the otelcol.receiver.otlp component, Alloy can listen on standard gRPC and HTTP ports to receive spans sent by your instrumented applications. It is also capable of receiving traces from legacy systems using Jaeger or Zipkin protocols, thus centralizing all trace collection.
2. Transformation and Sampling
Adapting spans and sampling.
Optimizing trace storage
The volume of data generated by distributed tracing can be colossal. Alloy's processor components (otelcol.processor) allow you to transform data in flight: modify span attributes, mask sensitive information (PII), and importantly, apply sampling strategies. You can configure probabilistic sampling to keep only a certain percentage of traces, or use tail-based sampling to specifically keep traces with errors or that are particularly slow, before sending them to Grafana Tempo via otelcol.exporter.otlp.
Best Practice: Use the otelcol.processor.tail_sampling component to implement intelligent sampling. Keep 100% of traces containing errors or abnormal latency, but only 5% of normal success traces. This significantly reduces your Tempo costs.Common Mistake: Sending the entirety (100%) of traces from a high-traffic application directly to Tempo. This causes network saturation, very high costs, and overall slow exploration.
otelcol.processor.tail_sampling "smart_sampler" {
policy {
name = "keep-errors"
type = "status_code"
status_code {
status_codes = ["ERROR"]
}
}
policy {
name = "probabilistic-success"
type = "probabilistic"
probabilistic {
sampling_percentage = 5
}
}
output {
metrics = [otelcol.exporter.otlp.tempo.input]
}
}