Transformation

2. Transformation and Sampling #

Adapting spans and sampling.

Optimizing trace storage #

The volume of data generated by distributed tracing can be colossal. Alloy's processor components (otelcol.processor) allow you to transform data in flight: modify span attributes, mask sensitive information (PII), and importantly, apply sampling strategies. You can configure probabilistic sampling to keep only a certain percentage of traces, or use tail-based sampling to specifically keep traces with errors or that are particularly slow, before sending them to Grafana Tempo via otelcol.exporter.otlp.

Best Practice: Use the otelcol.processor.tail_sampling component to implement intelligent sampling. Keep 100% of traces containing errors or abnormal latency, but only 5% of normal success traces. This significantly reduces your Tempo costs.

Common Mistake: Sending the entirety (100%) of traces from a high-traffic application directly to Tempo. This causes network saturation, very high costs, and overall slow exploration.

otelcol.processor.tail_sampling "smart_sampler" {
  policy {
    name = "keep-errors"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }
  policy {
    name = "probabilistic-success"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 5
    }
  }
  output {
    metrics = [otelcol.exporter.otlp.tempo.input]
  }
}