Thanos Store Gateway and Compactor: history, downsampling and retention

1. Store Gateway #

Reading history from object storage.

Starting the store #

thanos store \
  --data-dir=/var/thanos/store \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --index-cache-size=2GB \
  --chunk-pool-size=2GB \
  --store.limits.request-series=500000 \
  --min-time=-90d      # ne servir que les 90 derniers jours (partitionnement temporel)

# Docker Compose (extrait)
  store:
    image: quay.io/thanos/thanos:latest
    command:
      - store
      - --data-dir=/var/thanos/store
      - --objstore.config-file=/etc/thanos/bucket.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
    volumes: ["./bucket.yml:/etc/thanos/bucket.yml", "store-data:/var/thanos/store"]

Good to know: The Store Gateway keeps only the lightweight index and headers on disk: data is read on demand from object storage. Provide a persistent volume to avoid downloading everything again at each restart.

2. Compactor #

Compacting, downsampling, deleting.

The three missions #

• Compaction: merges small 2h blocks into larger blocks (up to 14 days) and deduplicates replicas vertically.
• Downsampling: creates 5-minute then 1-hour blocks (min, max, sum, count, counter aggregates).
• Retention: deletes blocks older than the configured duration, per resolution.

thanos compact \
  --data-dir=/var/thanos/compact \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --http-address=0.0.0.0:10902 \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=180d \
  --retention.resolution-1h=730d \
  --compact.concurrency=2 \
  --downsample.concurrency=2 \
  --delete-delay=48h \
  --wait                  # tourne en continu (sinon une passe puis arrêt)

Common mistake: Only one compactor instance must write to a given bucket. Two simultaneous compactors corrupt blocks. Watch thanos_compact_halted: at 1, the compactor has stopped on a critical error.

3. Bucket tools #

Inspecting and repairing.

thanos tools bucket #

thanos tools bucket ls       --objstore.config-file=bucket.yml            # lister les blocs
thanos tools bucket inspect  --objstore.config-file=bucket.yml            # tableau : ULID, plage, résolution, séries, taille
thanos tools bucket verify   --objstore.config-file=bucket.yml            # vérifier l'intégrité
thanos tools bucket web      --objstore.config-file=bucket.yml            # interface web des blocs
thanos tools bucket downsample --objstore.config-file=bucket.yml          # forcer un downsampling
thanos tools bucket mark --id=<ULID> --marker=deletion-mark.json --details="bloc corrompu" --objstore.config-file=bucket.yml

4. Checking the result #

What does the bucket contain?

Sample output #

# Métriques utiles
thanos_compact_group_compactions_total
thanos_compact_downsample_total
thanos_compact_todo_compactions
thanos_blocks_meta_synced{state="loaded"}