Tempo: object storage, blocks, compactor and distributed deployment

1. Object storage #

Configuring S3 or MinIO.

storage #

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio:9000
      access_key: ${MINIO_USER}
      secret_key: ${MINIO_PASSWORD}
      insecure: true
    wal:
      path: /var/tempo/wal
    block:
      version: vParquet4
    pool:
      max_workers: 100
      queue_depth: 10000
    cache: memcached
    memcached:
      host: memcached
      service: memcached-client
      consistent_hash: true

docker run -d --name minio -p 9000:9000 -p 9001:9001 -e MINIO_ROOT_USER=minio -e MINIO_ROOT_PASSWORD=minio123 \
  quay.io/minio/minio server /data --console-address ":9001"
docker run --rm --network host --entrypoint sh quay.io/minio/mc -c "mc alias set l http://localhost:9000 minio minio123 && mc mb l/tempo-traces"
tempo -config.file=tempo.yaml -config.expand-env=true

2. Retention and compaction #

Controlling duration and volume.

Compactor #

compactor:
  compaction:
    block_retention: 336h            # 14 jours
    compacted_block_retention: 1h
    compaction_window: 1h
    max_compaction_objects: 6000000
  ring:
    kvstore: { store: memberlist }

# Outils de diagnostic des blocs
tempo-cli list blocks single-tenant --backend s3 --bucket tempo-traces --s3-endpoint minio:9000 --s3-insecure
tempo-cli query api <HOST> <TRACE_ID>             # récupérer une trace par l'API
curl -s http://localhost:3200/metrics | grep tempodb_blocklist_length

Common mistake: Without a compactor, the number of small blocks explodes and searches slow down. A single compactor instance per tenant is usually enough.

3. Scaling up #

tempo-distributed.

Distributed Helm #

# values-prod.yaml (extrait)
traces:
  otlp: { grpc: { enabled: true }, http: { enabled: true } }
storage:
  trace:
    backend: s3
    s3: { bucket: tempo-traces, endpoint: s3.eu-west-3.amazonaws.com, region: eu-west-3 }
ingester:    { replicas: 3, persistence: { enabled: true, size: 20Gi } }
distributor: { replicas: 3 }
querier:     { replicas: 3 }
queryFrontend: { replicas: 2 }
compactor:   { replicas: 1 }
metricsGenerator: { enabled: true }
global_overrides:
  defaults:
    ingestion: { rate_limit_bytes: 20000000, burst_size_bytes: 30000000 }

helm upgrade --install tempo grafana/tempo-distributed -n tracing --create-namespace -f values-prod.yaml
kubectl -n tracing get pods -o wide
kubectl -n tracing top pods

Sizing #