2. Transformation and Sampling #
Adapting spans and sampling.
Cleaning attributes #
A trace can contain sensitive data. otelcol.processor.attributes removes or masks it before it leaves your network.
otelcol.processor.attributes "nettoyage" {
action {
key = "http.request.header.authorization"
action = "delete"
}
action {
key = "user.email"
action = "hash"
}
action {
key = "deployment.environment.name"
value = "prod"
action = "upsert"
}
output {
traces = []
}
}| Received attribute | Sent attribute (measured) |
|---|---|
http.request.header.authorization: Bearer secret | Deleted |
user.email: alice@exemple.fr | 182e2bdf10871ea9… (hashed) |
| Missing | deployment.environment.name: prod |
Hashing keeps a stable value: you can still count affected users without knowing their address.
Normalizing with OTTL #
otelcol.processor.transform runs OTTL statements. Here we group GET /produits/42, GET /produits/43… under a single name: otherwise each ID creates a separate operation and dashboards become unreadable.
otelcol.processor.transform "noms" {
error_mode = "ignore"
trace_statements {
context = "span"
statements = [
`set(attributes["http.route"], "/produits/{id}") where IsMatch(name, "^GET /produits/[0-9]+$")`,
`set(name, "GET /produits/{id}") where IsMatch(name, "^GET /produits/[0-9]+$")`,
]
}
output {
traces = []
}
}Observed result: the span is called GET /produits/{id} and carries http.route=/produits/{id}. Order matters: write the attribute before renaming, or the condition no longer matches.
Sampling intelligently #
Keeping all traces is expensive. There are two approaches:
Head (probabilistic_sampler) | Tail (tail_sampling) | |
|---|---|---|
| Decision | At the first span | After seeing the trace |
| Keeps errors? | No, it is random | Yes |
| Cost | Very low | Memory: traces are held |
otelcol.processor.tail_sampling "decision" {
decision_wait = "30s"
num_traces = 50000
policy {
name = "garder-les-erreurs"
type = "status_code"
status_code {
status_codes = ["ERROR"]
}
}
policy {
name = "garder-les-lentes"
type = "latency"
latency {
threshold_ms = 1000
}
}
policy {
name = "echantillon-du-reste"
type = "probabilistic"
probabilistic {
sampling_percentage = 5
}
}
output {
traces = []
}
}A trace is kept as soon as one policy accepts it. We tested errors and latency (with decision_wait reduced to 5 s) on 3 traces:
| Trace | Result |
|---|---|
GET /produits/42 failing, 20 ms | Kept |
GET /sante successful, 10 ms | Discarded |
POST /payer successful, 2500 ms | Kept |
otelcol_receiver_accepted_spans_total was 3, and otelcol_exporter_sent_spans_total 2.
With several instances: all spans of a trace must reach the same instance. In front of several Alloy instances, addotelcol.exporter.loadbalancingwithrouting_key = "traceID".decision_wait(30 s by default) andnum_traces(50000) determine the memory needed.
Producing metrics from traces #
To keep real rates despite sampling, generate metrics before it: forward received traces both to this connector and to the pipeline.
otelcol.connector.spanmetrics "red" {
histogram {
explicit {
buckets = ["100ms", "500ms", "2s"]
}
}
output {
metrics = [otelcol.exporter.prometheus.vers_prom.input]
}
}
otelcol.exporter.prometheus "vers_prom" {
forward_to = [prometheus.remote_write.mimir.receiver]
}
prometheus.remote_write "mimir" {
endpoint {
url = "http://mimir:9009/api/v1/push"
}
}My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Alloy : recevoir de l'OTLP et l'envoyer vers Tempo et Mimir✓ solved · 1 reply(ies)
Ask a question about this lesson →See all discussions about this course
Course Glossary
Grafana Alloy
An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).
Recommended Articles
Grafana Alloy: collecting and transforming telemetry
Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...
Grafana Tempo: the distributed tracing backend at scale
Understand distributed tracing and Grafana Tempo: traces, spans, TraceID, architecture (distribut...
The LGTM stack (Loki, Grafana, Tempo, Mimir): architecture and setup
Grafana Labs' LGTM stack: Loki for logs, Grafana for visualization, Tempo for traces, Mimir for m...
Grafana Alloy: collecting system metrics and local logs
Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...
Grafana Alloy: The importance of Self-Monitoring
Discover why and how to configure Grafana Alloy so that it monitors itself, collecting its own lo...
Alloy and the Grafana Labs ecosystem: which component for which tool?
The map of Alloy integrations with Faro, Beyla, k6, Loki, Mimir, Tempo, Pyroscope, Database Obser...
Sending traces to Tempo: OpenTelemetry instrumentation and Alloy
Produce and route traces to Tempo: Python/Java/Node auto-instrumentation, the manual SDK, OTEL en...
Grafana Alloy: Log Management with Loki
Discover how to configure Grafana Alloy to read log files, journald, or network streams, process ...
Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry
Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...