RousselTMLEARNING

Article · Alloy: From Discovery to Expertise

Transformation

2. Transformation and Sampling #

Adapting spans and sampling.

Cleaning attributes #

A trace can contain sensitive data. otelcol.processor.attributes removes or masks it before it leaves your network.

otelcol.processor.attributes "nettoyage" {
  action {
    key    = "http.request.header.authorization"
    action = "delete"
  }
  action {
    key    = "user.email"
    action = "hash"
  }
  action {
    key    = "deployment.environment.name"
    value  = "prod"
    action = "upsert"
  }
  output {
    traces = []
  }
}
Received attributeSent attribute (measured)
http.request.header.authorization: Bearer secretDeleted
user.email: alice@exemple.fr182e2bdf10871ea9… (hashed)
Missingdeployment.environment.name: prod

Hashing keeps a stable value: you can still count affected users without knowing their address.

Normalizing with OTTL #

otelcol.processor.transform runs OTTL statements. Here we group GET /produits/42, GET /produits/43… under a single name: otherwise each ID creates a separate operation and dashboards become unreadable.

otelcol.processor.transform "noms" {
  error_mode = "ignore"
  trace_statements {
    context    = "span"
    statements = [
      `set(attributes["http.route"], "/produits/{id}") where IsMatch(name, "^GET /produits/[0-9]+$")`,
      `set(name, "GET /produits/{id}") where IsMatch(name, "^GET /produits/[0-9]+$")`,
    ]
  }
  output {
    traces = []
  }
}

Observed result: the span is called GET /produits/{id} and carries http.route=/produits/{id}. Order matters: write the attribute before renaming, or the condition no longer matches.

Sampling intelligently #

Keeping all traces is expensive. There are two approaches:

Head (probabilistic_sampler)Tail (tail_sampling)
DecisionAt the first spanAfter seeing the trace
Keeps errors?No, it is randomYes
CostVery lowMemory: traces are held
otelcol.processor.tail_sampling "decision" {
  decision_wait = "30s"
  num_traces    = 50000

  policy {
    name = "garder-les-erreurs"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }

  policy {
    name = "garder-les-lentes"
    type = "latency"
    latency {
      threshold_ms = 1000
    }
  }

  policy {
    name = "echantillon-du-reste"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 5
    }
  }

  output {
    traces = []
  }
}

A trace is kept as soon as one policy accepts it. We tested errors and latency (with decision_wait reduced to 5 s) on 3 traces:

TraceResult
GET /produits/42 failing, 20 msKept
GET /sante successful, 10 msDiscarded
POST /payer successful, 2500 msKept

otelcol_receiver_accepted_spans_total was 3, and otelcol_exporter_sent_spans_total 2.

With several instances: all spans of a trace must reach the same instance. In front of several Alloy instances, add otelcol.exporter.loadbalancing with routing_key = "traceID". decision_wait (30 s by default) and num_traces (50000) determine the memory needed.

Producing metrics from traces #

To keep real rates despite sampling, generate metrics before it: forward received traces both to this connector and to the pipeline.

otelcol.connector.spanmetrics "red" {
  histogram {
    explicit {
      buckets = ["100ms", "500ms", "2s"]
    }
  }
  output {
    metrics = [otelcol.exporter.prometheus.vers_prom.input]
  }
}

otelcol.exporter.prometheus "vers_prom" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

📄 Grafana Alloy: Trace Management with Tempo →

My progress

  • ⏱ Estimated time : ≈ 12 min

Sign in to track your progress and take notes. Sign in

⭐ Rate this lesson

5.0 initial rating from the teaching team

Sign in to rate this lesson →

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Course Glossary

Grafana Alloy

An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).

Recommended Articles

Grafana Alloy: collecting and transforming telemetry

Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...

Grafana Tempo: the distributed tracing backend at scale

Understand distributed tracing and Grafana Tempo: traces, spans, TraceID, architecture (distribut...

The LGTM stack (Loki, Grafana, Tempo, Mimir): architecture and setup

Grafana Labs' LGTM stack: Loki for logs, Grafana for visualization, Tempo for traces, Mimir for m...

Grafana Alloy: collecting system metrics and local logs

Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...

Grafana Alloy: The importance of Self-Monitoring

Discover why and how to configure Grafana Alloy so that it monitors itself, collecting its own lo...

Alloy and the Grafana Labs ecosystem: which component for which tool?

The map of Alloy integrations with Faro, Beyla, k6, Loki, Mimir, Tempo, Pyroscope, Database Obser...

Sending traces to Tempo: OpenTelemetry instrumentation and Alloy

Produce and route traces to Tempo: Python/Java/Node auto-instrumentation, the manual SDK, OTEL en...

Grafana Alloy: Log Management with Loki

Discover how to configure Grafana Alloy to read log files, journald, or network streams, process ...

Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry

Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!