1. Monitoring Thanos #
The essential signals.
Key metrics and alerts #
| Component | Metric | Alert if... |
|---|---|---|
| Sidecar | thanos_shipper_uploads_failed_total | Increasing |
| Sidecar | thanos_sidecar_prometheus_up | = 0 |
| Query | thanos_store_nodes_grpc_connections | Fewer stores than expected |
| Query | grpc_server_handled_total{grpc_code!="OK"} | High error rate |
| Store | thanos_bucket_store_series_blocks_queried | Very high (expensive queries) |
| Compactor | thanos_compact_halted | = 1 |
| Compactor | thanos_compact_todo_compactions | Growing endlessly |
| Ruler | prometheus_rule_evaluation_failures_total | > 0 |
# Dashboards et alertes officiels
git clone https://github.com/thanos-io/thanos && cd thanos/mixin
jb install && make build
# Alertes minimales
- alert: ThanosCompactHalted
expr: thanos_compact_halted == 1
for: 5m
labels: { severity: critical }
- alert: ThanosSidecarNoUpload
expr: increase(thanos_shipper_uploads_total[6h]) == 0
for: 30m
labels: { severity: warning }2. Performance #
Fast queries over long periods.
Downsampling and caches #
--query.auto-downsampling, Query picks the resolution suited to the graph step.# Query : résolution automatique + limites de protection
thanos query --query.auto-downsampling --query.max-concurrent=20 --query.timeout=2m \
--store.response-timeout=30s --query.default-evaluation-interval=1m
# Store : caches
thanos store --index-cache.config-file=index-cache.yml --store.caching-bucket.config-file=caching-bucket.yml
# index-cache.yml
type: MEMCACHED
config:
addresses: [memcached:11211]
max_async_buffer_size: 10000Good practice: Use recording rules for heavy dashboard queries: the Ruler or Prometheus precomputes them, which avoids reading large historical volumes at each refresh.
3. Troubleshooting #
Frequent errors.
Diagnostic table #
| Symptom | Likely cause | Action |
|---|---|---|
| No blocks in the bucket | Local compaction enabled or sidecar without access to the TSDB | Check the 2h block flags and the shared volume |
| Duplicate series in Grafana | Replica label not configured | Add --query.replica-label |
| « expanding series: context deadline exceeded » | Query too broad | Reduce the range, use downsampling, increase timeouts |
| Missing data over a period | Store Gateway not up to date or --min-time too recent | Check thanos_blocks_meta_synced |
| Compactor halted | Overlapping / corrupted blocks | thanos tools bucket verify --repair after analysis |
| High store memory usage | Too many series returned | Limit --store.limits.request-series and add cache |
thanos tools bucket verify --objstore.config-file=bucket.yml --issues=overlapped_blocks,index_known_issues
curl -s localhost:10902/-/healthy; curl -s localhost:10902/-/ready
journalctl -u thanos-compact -n 100 --no-pager | grep -i error4. Thanos versus the alternatives #
Making an informed choice.
Comparison #
| Solution | Model | Strengths | Points of attention |
|---|---|---|---|
| Thanos | Components around Prometheus (sidecar / receive) | Gradual adoption, simple, CNCF community | Several components to operate |
| Grafana Mimir | Distributed multi-tenant TSDB (remote write) | Very large scale, performance, ease of use | Replaces local storage, more resources |
| Cortex | Mimir's ancestor | Proven | Mimir is its recommended successor |
| VictoriaMetrics | Independent PromQL-compatible TSDB | Very resource-efficient | Different ecosystem and licenses |
Good practice: Thanos is ideal if you already have Prometheus servers to keep and want to add retention, HA and a global view without migrating everything. For a very large-scale multi-tenant platform, evaluate Mimir.
