Mimir: the write path (distributor, ingesters, replication and HA deduplication)

1. Sending metrics to Mimir #

Remote write, OTLP and tenant.

Configuring Prometheus #

Prometheus keeps collecting (or switches to agent mode, without local storage or queries) and pushes its samples to Mimir.

# prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    cluster: prod-eu      # identifie la source
    __replica__: replica-a  # identifie le réplica HA (retiré par Mimir)

remote_write:
  - url: http://mimir:8080/api/v1/push
    headers:
      X-Scope-OrgID: team-platform   # tenant
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
      max_shards: 50

Good to know: With multitenancy_enabled: false, the X-Scope-OrgID header is not needed: all data goes to the anonymous tenant.

With Grafana Alloy #

prometheus.remote_write "mimir" {
  endpoint {
    url     = "http://mimir:8080/api/v1/push"
    headers = { "X-Scope-OrgID" = "team-platform" }
  }
}

2. Distributor and ingesters #

Validate, distribute, replicate.

The journey of a sample #

1. The distributor checks the tenant, label validity (number, length, names) and timestamp, and applies ingestion limits. Exceeding them returns 429 (rate limit) or 400 (invalid sample).
2. It hashes tenant + labels to find, in the ring, the 3 ingesters (default replication factor) responsible for the series.
3. Each ingester adds the sample to its in-memory TSDB and writes it to its WAL (write-ahead log) on disk, to survive a restart.
4. Every 2 hours, the ingester closes a TSDB block and uploads it to object storage. It keeps recent blocks locally to answer queries (13h by default).

Replication and quorum #

With a replication factor of 3, the write is confirmed as soon as 2 ingesters out of 3 have responded. You can therefore lose an ingester (or an entire zone with zone-aware replication) without data loss or client-side errors.

ingester:
  ring:
    replication_factor: 3
    zone_awareness_enabled: true    # une copie par zone de disponibilité
    instance_availability_zone: zone-a

Common mistake: Ingesters are stateful: deploy them as a StatefulSet with a persistent volume for the WAL, and restart them one at a time (or one zone at a time). Deleting several ingesters at once can lose data that has not been uploaded yet.

3. Deduplicating Prometheus HA pairs #

Two replicas, a single stored series.

Principle #

For high availability, two identical Prometheus servers are run. Without precautions, Mimir would store each series twice. The distributor's HA tracker elects, for each (tenant, cluster) pair, a « leader » replica and rejects the other one's samples. If the leader goes silent for longer than ha_tracker_failover_timeout (30 s by default), the other takes over.

# mimir.yaml
limits:
  accept_ha_samples: true
  ha_cluster_label: cluster        # valeurs par défaut
  ha_replica_label: __replica__

distributor:
  ha_tracker:
    enable_ha_tracker: true
    kvstore:
      store: consul
      consul:
        host: consul:8500

Good practice: Both Prometheus servers must have the same value for cluster and a different value for __replica__. Mimir removes the replica label before storage.

The /distributor/ha_tracker page shows the elected replica for each cluster. The cortex_distributor_deduped_samples_total metric counts the discarded samples.