3. Remote Write #
Forwarding data to Grafana Cloud or a local instance.
Sending to Mimir, Prometheus or Grafana Cloud #
prometheus.remote_write sends metrics to any compatible backend. It first writes to an on-disk log (WAL), then ships from that log: a backend outage therefore does not lose data immediately.
prometheus.remote_write "mimir" {
external_labels = { region = "eu-west" }
endpoint {
url = "https://mimir.example.com/api/v1/push"
basic_auth {
username = sys.env("MIMIR_USER")
password = sys.env("MIMIR_PASSWORD")
}
queue_config {
capacity = 10000
max_shards = 10
max_samples_per_send = 2000
}
write_relabel_config {
source_labels = ["__name__"]
regex = "up|node_.*"
action = "keep"
}
}
wal {
truncate_frequency = "2h"
max_keepalive_time = "8h"
}
}Useful settings #
| Setting | Purpose | Default |
|---|---|---|
external_labels | Labels added to everything sent | – |
queue_config.capacity | Samples buffered per shard | 10000 |
queue_config.max_shards | Maximum concurrent sends | 50 |
queue_config.max_samples_per_send | Size of one send | 2000 |
wal.truncate_frequency | WAL cleanup frequency | 2h |
wal.max_keepalive_time | Maximum retention when sending fails | 8h |
During an outage: data is kept about 8 hours by default, and the disk grows meanwhile. Plan space for --storage.path.Filtering per destination #
write_relabel_config filters for one backend only. We measured it by sending the same data to two destinations: the one keeping only up and alloy_build_info received 8 samples, the other 1765. This is the simplest way to reduce a storage bill.
Monitoring sending #
# Samples failing to send (alert on this)
rate(prometheus_remote_storage_samples_failed_total[5m]) > 0
# Queue towards the backend
prometheus_remote_storage_samples_pending
# Scraped targets
prometheus_scrape_targets_gaugeBest practice: keep passwords in the environment or a file (see Secrets Management), and make --storage.path persistent: otherwise the WAL disappears when the container restarts.My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Alloy : recevoir de l'OTLP et l'envoyer vers Tempo et Mimir✓ solved · 1 reply(ies)
Ask a question about this lesson →See all discussions about this course
Course Glossary
Grafana Alloy
An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).
Recommended Articles
Ingesting metrics into VictoriaMetrics: scraping, vmagent and remote write
All the ways to feed VictoriaMetrics: remote write from Prometheus, built-in scraping, vmagent an...
Grafana Alloy: collecting system metrics and local logs
Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...
Sending logs, metrics and traces to OpenObserve
OpenObserve receiving endpoints: JSON, bulk, OTLP with the OpenTelemetry Collector, Prometheus re...
Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry
Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...
Grafana Alloy: collecting and transforming telemetry
Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...
k6 and Alloy: sending load test results over OpenTelemetry
Send k6 metrics to Alloy with the OpenTelemetry output, convert them for Mimir and compare them w...
Grafana Alloy vs OpenTelemetry Collector: Which One Should You Choose?
A detailed comparison between Grafana Alloy and the OpenTelemetry Collector. Discover the strengt...
Alloy and the Grafana Labs ecosystem: which component for which tool?
The map of Alloy integrations with Faro, Beyla, k6, Loki, Mimir, Tempo, Pyroscope, Database Obser...
Thanos Receive: ingesting remote write and handling multi-tenancy
Use Thanos Receive: push mode with remote write, hashring, replication, multi-tenancy, Prometheus...