OpenTelemetry Collector: processors (transforming, filtering and protecting telemetry)

1. The role of processors #

Between reception and export.

What do they do? #

A processor modifies, filters or groups telemetry (traces, metrics, logs) between a receiver and an exporter. Processors run in the order they are declared in the pipeline: the order is therefore significant.

# Le pipeline relie receivers -> processors (dans l'ordre) -> exporters
service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, k8sattributes, resource, filter, transform, batch]
      exporters:  [otlphttp/tempo]

2. The essential processors #

Stability and performance.

memory_limiter and batch #

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80          # refuse les données au-delà de 80 % de la mémoire disponible
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 10000

Good practice: Without memory_limiter, a telemetry spike gets the Collector killed (OOMKilled) and loses everything in memory. With it, the Collector temporarily refuses data and the sender retries.

attributes and resource #

processors:
  resource:
    attributes:
      - { key: deployment.environment, value: prod, action: upsert }
      - { key: host.name, action: delete }
  attributes/redact:
    actions:
      - { key: http.request.header.authorization, action: delete }
      - { key: user.email, action: hash }
      - { key: db.statement, action: update, value: "[masqué]" }
      - { key: credit_card, pattern: "\\d{12,16}", action: extract }

Common mistake: Delete or hash personal data in the Collector: it is the last checkpoint before storage (GDPR).

3. Filtering and transforming with OTTL #

OpenTelemetry's transformation language.

filter and transform #

processors:
  filter/healthchecks:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.target"] == "/healthz"'
        - 'IsMatch(attributes["http.target"], "^/(ready|live).*")'
    metrics:
      metric:
        - 'name == "http.server.duration" and resource.attributes["service.name"] == "legacy"'

  transform/normalize:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - set(attributes["env"], resource.attributes["deployment.environment"])
          - replace_pattern(attributes["http.url"], "token=[^&]+", "token=***")
          - truncate_all(attributes, 256)
    log_statements:
      - context: log
        statements:
          - set(severity_text, "ERROR") where body == "panic"
          - set(attributes["team"], "paiement") where resource.attributes["service.name"] == "checkout"

Enriching in Kubernetes #

processors:
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.node.name, k8s.deployment.name]
      labels:
        - { tag_name: app, key: app.kubernetes.io/name, from: pod }
    pod_association:
      - sources: [{ from: resource_attribute, name: k8s.pod.ip }]
      - sources: [{ from: connection }]

Good to know: The Collector must run with a ServiceAccount allowed to read pods, namespaces and replicasets (ClusterRole), otherwise enrichment silently fails.

4. Sampling and Alloy equivalents #

Reducing volume.

tail_sampling #

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - { name: erreurs, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: lentes,  type: latency,     latency: { threshold_ms: 1000 } }
      - { name: reste,   type: probabilistic, probabilistic: { sampling_percentage: 5 } }

Validating and testing #

otelcol-contrib validate --config=config.yaml
otelcol-contrib components | grep -A20 processors          # processors disponibles dans ce build

# Voir ce que devient la télémétrie : exporter debug temporaire
exporters: { debug: { verbosity: detailed } }

docker run --rm --network host ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest \
  traces --otlp-insecure --traces 5 --otlp-attributes 'user.email="a@b.fr"'

// Équivalent Grafana Alloy
otelcol.processor.memory_limiter "default" { check_interval = "1s"  limit_percentage = 80  output { traces = [otelcol.processor.batch.default.input] } }
otelcol.processor.batch "default" { output { traces = [otelcol.exporter.otlphttp.tempo.input] } }
otelcol.processor.attributes "redact" {
  action { key = "user.email"  action = "hash" }
  output { traces = [otelcol.processor.batch.default.input] }
}