Mimir: compactor, retention and object storage

1. Configuring object storage #

Three separate buckets.

Common configuration #

Mimir stores three things in object storage: metric blocks, ruler rules and Alertmanager configuration. Use three different buckets (or three prefixes).

common:
  storage:
    backend: s3
    s3:
      endpoint: minio:9000
      access_key_id: mimir
      secret_access_key: supersecret
      insecure: true

blocks_storage:
  s3: { bucket_name: mimir-blocks }
  tsdb: { dir: /data/tsdb }
  bucket_store: { sync_dir: /data/tsdb-sync }

ruler_storage:
  s3: { bucket_name: mimir-ruler }

alertmanager_storage:
  s3: { bucket_name: mimir-alertmanager }

Common mistake: Mimir refuses to start if blocks and rules share the same bucket without a prefix: their directory trees would get mixed up.

Good to know: Available backends: s3 (AWS and compatible: MinIO, Ceph, Scaleway, OVH...), gcs, azure, swift and filesystem (tests only).

2. Compaction #

Fewer blocks, fewer duplicates, faster queries.

Why compact? #

Each ingester produces a block every 2h; with 30 ingesters and a replication factor of 3, that makes hundreds of small blocks, each containing duplicates. The compactor:

• merges blocks of the same period vertically (and deduplicates replicated samples);
• merges adjacent periods horizontally (2h → 12h → 24h);
• splits very large tenants into several blocks (split-and-merge), to parallelize;
• updates the bucket index, then marks and deletes replaced or expired blocks.

Settings #

compactor:
  data_dir: /data/compactor
  block_ranges: [2h, 12h, 24h]
  compaction_interval: 1h
  deletion_delay: 12h            # délai avant suppression physique d'un bloc remplacé
  sharding_ring:
    kvstore: { store: memberlist }

limits:
  compactor_split_and_merge_shards: 0   # > 0 pour les très gros tenants
  compactor_split_groups: 1

Good practice: Watch cortex_compactor_last_successful_run_timestamp_seconds: a stuck compactor does not cause a visible error right away, but queries slow down and storage grows.

3. Retention and deletion #

Keeping what is needed, for as long as needed.

Retention #

Retention is applied by the compactor, globally or per tenant. By default it is infinite (0).

limits:
  compactor_blocks_retention_period: 1y

# runtime.yaml : surcharges par tenant
overrides:
  team-dev:
    compactor_blocks_retention_period: 30d
  team-finance:
    compactor_blocks_retention_period: 5y

Good to know: Mimir does not do downsampling (unlike Thanos). Long retention remains reasonable thanks to the price of object storage, but queries over several months benefit greatly from recording rules that pre-aggregate the data.

Deleting series #

Mimir offers no targeted deletion of series: data sent by mistake disappears with retention. So filter sensitive labels upstream (write_relabel_configs). To delete an entire tenant, use the /compactor/delete_tenant API.

curl -X POST -H 'X-Scope-OrgID: team-old' http://mimir:8080/compactor/delete_tenant
curl -H 'X-Scope-OrgID: team-old' http://mimir:8080/compactor/delete_tenant_status