RousselTMLEARNING

Article · Alloy: From Discovery to Expertise

Remote Write

3. Remote Write #

Forwarding data to Grafana Cloud or a local instance.

Sending to Mimir, Prometheus or Grafana Cloud #

prometheus.remote_write sends metrics to any compatible backend. It first writes to an on-disk log (WAL), then ships from that log: a backend outage therefore does not lose data immediately.

prometheus.remote_write "mimir" {
  external_labels = { region = "eu-west" }

  endpoint {
    url = "https://mimir.example.com/api/v1/push"

    basic_auth {
      username = sys.env("MIMIR_USER")
      password = sys.env("MIMIR_PASSWORD")
    }

    queue_config {
      capacity             = 10000
      max_shards           = 10
      max_samples_per_send = 2000
    }

    write_relabel_config {
      source_labels = ["__name__"]
      regex         = "up|node_.*"
      action        = "keep"
    }
  }

  wal {
    truncate_frequency = "2h"
    max_keepalive_time = "8h"
  }
}

Useful settings #

SettingPurposeDefault
external_labelsLabels added to everything sent–
queue_config.capacitySamples buffered per shard10000
queue_config.max_shardsMaximum concurrent sends50
queue_config.max_samples_per_sendSize of one send2000
wal.truncate_frequencyWAL cleanup frequency2h
wal.max_keepalive_timeMaximum retention when sending fails8h
During an outage: data is kept about 8 hours by default, and the disk grows meanwhile. Plan space for --storage.path.

Filtering per destination #

write_relabel_config filters for one backend only. We measured it by sending the same data to two destinations: the one keeping only up and alloy_build_info received 8 samples, the other 1765. This is the simplest way to reduce a storage bill.

Monitoring sending #

# Samples failing to send (alert on this)
rate(prometheus_remote_storage_samples_failed_total[5m]) > 0

# Queue towards the backend
prometheus_remote_storage_samples_pending

# Scraped targets
prometheus_scrape_targets_gauge
Best practice: keep passwords in the environment or a file (see Secrets Management), and make --storage.path persistent: otherwise the WAL disappears when the container restarts.

📄 Grafana Alloy: Metrics Collection (Prometheus & Ecosystem) →

My progress

  • ⏱ Estimated time : ≈ 6 min

Sign in to track your progress and take notes. Sign in

⭐ Rate this lesson

5.0 initial rating from the teaching team

Sign in to rate this lesson →

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Course Glossary

Grafana Alloy

An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).

Recommended Articles

Ingesting metrics into VictoriaMetrics: scraping, vmagent and remote write

All the ways to feed VictoriaMetrics: remote write from Prometheus, built-in scraping, vmagent an...

Grafana Alloy: collecting system metrics and local logs

Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...

Sending logs, metrics and traces to OpenObserve

OpenObserve receiving endpoints: JSON, bulk, OTLP with the OpenTelemetry Collector, Prometheus re...

Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry

Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...

Grafana Alloy: collecting and transforming telemetry

Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...

k6 and Alloy: sending load test results over OpenTelemetry

Send k6 metrics to Alloy with the OpenTelemetry output, convert them for Mimir and compare them w...

Grafana Alloy vs OpenTelemetry Collector: Which One Should You Choose?

A detailed comparison between Grafana Alloy and the OpenTelemetry Collector. Discover the strengt...

Alloy and the Grafana Labs ecosystem: which component for which tool?

The map of Alloy integrations with Faro, Beyla, k6, Loki, Mimir, Tempo, Pyroscope, Database Obser...

Thanos Receive: ingesting remote write and handling multi-tenancy

Use Thanos Receive: push mode with remote write, hashring, replication, multi-tenancy, Prometheus...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!