Thanos Sidecar: attaching Thanos to Prometheus and uploading to object storage

1. Preparing Prometheus #

The essential prerequisites.

Mandatory settings #

• External labels unique per Prometheus (cluster, replica): essential for deduplication and identification.
• Local compaction disabled: --storage.tsdb.min-block-duration=2h and --storage.tsdb.max-block-duration=2h (the Thanos compactor takes care of it).
• Short local retention (a few days): history lives in object storage.
• Admin API and lifecycle so that the sidecar can reload the configuration.

# prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    cluster: prod-eu
    replica: A

scrape_configs:
  - job_name: prometheus
    static_configs: [{ targets: ["localhost:9090"] }]
  - job_name: node
    static_configs: [{ targets: ["node-exporter:9100"] }]

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.min-block-duration=2h \
  --storage.tsdb.max-block-duration=2h \
  --storage.tsdb.retention.time=2d \
  --web.enable-lifecycle --web.enable-admin-api

Common mistake: Never leave two Prometheus servers with exactly the same external_labels apart from replica: Thanos could no longer tell the sources apart and would duplicate or wrongly merge blocks.

2. Starting the sidecar #

With its object storage file.

Configuration and command #

# bucket.yml (stockage objet partagé par tous les composants Thanos)
type: S3
config:
  bucket: thanos
  endpoint: minio:9000
  access_key: minio
  secret_key: minio123
  insecure: true

thanos sidecar \
  --tsdb.path=/prometheus \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902

3. Complete Docker Compose stack #

Prometheus, sidecar and MinIO.

docker-compose.yml #

services:
  minio:
    image: quay.io/minio/minio
    command: server /data --console-address ":9001"
    environment: { MINIO_ROOT_USER: minio, MINIO_ROOT_PASSWORD: minio123 }
    ports: ["9000:9000", "9001:9001"]
  minio-init:
    image: quay.io/minio/mc
    depends_on: [minio]
    entrypoint: ["sh", "-c", "sleep 5 && mc alias set l http://minio:9000 minio minio123 && mc mb -p l/thanos"]

  prometheus:
    image: prom/prometheus:latest
    user: root
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.min-block-duration=2h
      - --storage.tsdb.max-block-duration=2h
      - --web.enable-lifecycle
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prom-data:/prometheus
    ports: ["9090:9090"]

  sidecar:
    image: quay.io/thanos/thanos:latest
    command:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://prometheus:9090
      - --objstore.config-file=/etc/thanos/bucket.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
    volumes:
      - ./bucket.yml:/etc/thanos/bucket.yml
      - prom-data:/prometheus
    depends_on: [prometheus, minio]

volumes: { prom-data: {} }

4. Checking #

Has the block reached the bucket?

Checks #

# Attendre ~2 h (ou forcer un bloc : curl -X POST localhost:9090/api/v1/admin/tsdb/snapshot)
docker compose run --rm --entrypoint sh quay.io/minio/mc -c "mc alias set l http://minio:9000 minio minio123 && mc ls l/thanos/"

thanos tools bucket ls --objstore.config-file=bucket.yml
thanos tools bucket inspect --objstore.config-file=bucket.yml

curl -s localhost:10902/metrics | grep -E 'thanos_shipper_uploads_total|thanos_sidecar_prometheus_up'

Good practice: If thanos_shipper_uploads_total stays at 0, first check that local compaction is really disabled and that the sidecar sees the same /prometheus directory as Prometheus.