RousselTMLEARNING

Article · Alloy: From Discovery to Expertise

Alloy Observability

1. Introduction: Why monitor the monitor? #

Your telemetry collector is a critical component of your observability infrastructure. If it crashes or silently degrades, you lose all visibility into your applications. Self-monitoring involves treating Grafana Alloy like any other critical application: by monitoring its health, performance, and errors.

The risks of an unmonitored collector #

Available in: ✓ OSS ✓ Enterprise ✓ Cloud

  • Silent data loss: A misconfigured component may stop sending data without generating a visible error.
  • Excessive resource consumption: An inefficient log pipeline can lead to excessive CPU or RAM consumption, impacting other applications on the same host.
  • Configuration errors: Syntax errors in a .river file can prevent the configuration from reloading.

Self-monitoring allows detecting these issues proactively.

2. Collect Alloy metrics #

Alloy exposes its own health and performance metrics in Prometheus format, as well as host metrics.

Host metrics (CPU, RAM, Disk) #

The prometheus.exporter.unix component (similar to node_exporter) is ideal for collecting basic system metrics on the host where Alloy is installed.
// Expose system metrics (CPU, memory, disk, network)
prometheus.exporter.unix "local_system" {}

// Scrape the metrics exposed by the exporter above
prometheus.scrape "scrape_system" {
  targets    = prometheus.exporter.unix.local_system.targets
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  // ... your Mimir/Prometheus endpoint configuration
}

Alloy internal metrics #

Alloy exposes its own internal health and performance metrics.
WARNING: Do not use the UI (e.g., {"__address__" = "localhost:12345", "job" = "alloy"}) to collect metrics because it can be disabled. Instead, use prometheus.exporter.self to collect metrics.

Use the prometheus.exporter.self component and add it to your scraping targets.

// Expose Alloy's internal metrics
prometheus.exporter.self "alloy_internal" {}

prometheus.scrape "scrape_alloy_and_system" {
  targets = concat(
    prometheus.exporter.unix.local_system.targets,
    prometheus.exporter.self.alloy_internal.targets
  )
  forward_to = [prometheus.remote_write.mimir.receiver]
}

// Key metrics to monitor:
// - alloy_component_health: Health of each component (0=unhealthy, 1=healthy)
// - process_cpu_seconds_total: CPU time consumed by Alloy
// - process_resident_memory_bytes: RAM memory used by Alloy

3. Collect Alloy logs #

Capturing the logs produced by Alloy is essential for diagnosing configuration errors or runtime issues.

Recommended method: via journald #

If Alloy is run as a systemd service, the best method is to read the system journal directly.
// Discover and filter logs for 'alloy.service'
discovery.relabel "journal_filter" {
  rule {
    source_labels = ["__journal__systemd_unit"]
    regex         = "alloy\\.service"
    action        = "keep"
  }
}

// Read logs from journald
loki.source.journal "read_alloy_logs" {
  forward_to = [loki.write.loki_endpoint.receiver]
  relabel_rules = discovery.relabel.journal_filter.rules
}

loki.write "loki_endpoint" {
  // ... your Loki endpoint configuration
}

Alternative: via a local file #

If you redirect Alloy's standard output to a file (e.g., /var/log/alloy.log), you can use loki.source.file.
local.file_match "alloy_log_file" {
  path_targets = [{"__path__" = "/var/log/alloy.log"}]
}

loki.source.file "read_from_file" {
  targets    = local.file_match.alloy_log_file.targets
  forward_to = [loki.write.loki_endpoint.receiver]
}

Don't forget to configure log rotation to avoid filling up the disk.

4. Collect Traces and Profiles (Profiling) #

For advanced performance debugging, Alloy can expose performance profiles via its web interface.

Enable profiling with pprof #

Profiling endpoints (pprof) are available on Alloy's web interface.
WARNING: Just like with metrics, relying on the local UI (e.g., {"__address__" = "localhost:12345"}) for profiling can be risky if it is disabled in production. Ensure it is active if you use pyroscope.scrape.
pyroscope.scrape "alloy_profiling" {
  targets = [
    {"__address__" = "localhost:12345", "job" = "alloy"}
  ]
  forward_to = [pyroscope.write.pyroscope_endpoint.receiver]
}

pyroscope.write "pyroscope_endpoint" {
  endpoint {
    url = "http://pyroscope:4040/api/v1/push"
  }
}

// Collected profiles: 
// - alloy_process_cpu: CPU utilization profile
// - alloy_process_mem: Memory allocation (heap) profile

Collecting Alloy's internal traces is a very advanced use case, generally reserved for the product's development itself. For most users, metrics and profiles are sufficient for self-monitoring.

5. Conclusion: A loop of trust #

By configuring Grafana Alloy to monitor itself, you create a loop of trust. You can not only validate that your collector is working properly, but also optimize its performance and be alerted immediately in case of an issue. This is a fundamental step in building a robust and reliable observability platform.

2026 update: what has changed #

Recent changes worth knowing (checked in November 2026).

2026 additions #

Grafana Alloy ships a minor release roughly every three weeks; 1.21 (October 2026) is the latest at the time of this update. Read the release notes before each upgrade: some releases bring a new Prometheus or OpenTelemetry Collector branch, with flagged breaking changes.

  • The livedebugging block shows, in Alloy's UI, the data flowing through each component, with no restart: ideal to check a pipeline.
  • With Fleet Management (Grafana Cloud), each Alloy instance's health and configuration are tracked from one console.
  • Recommended alerts: unhealthy components (alloy_component_controller_running_components{health_type!="healthy"}), remote write send failures and a growing WAL queue.
// Alloy's internal metrics, sent with the others
prometheus.exporter.self "alloy" { }
prometheus.scrape "alloy" {
  targets    = prometheus.exporter.self.alloy.targets
  forward_to = [prometheus.remote_write.mimir.receiver]
}
// Live debugging in the UI (http://localhost:12345)
livedebugging { enabled = true }

📄 Grafana Alloy: The importance of Self-Monitoring →

My progress

  • ⏱ Estimated time : ≈ 15 min

Sign in to track your progress and take notes. Sign in

⭐ Rate this lesson

5.0 initial rating from the teaching team

Sign in to rate this lesson →

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Course Glossary

Grafana Alloy

An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).

Recommended Articles

Grafana Alloy: collecting system metrics and local logs

Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...

Grafana Alloy: Syntax and Configuration (Alloy Language: River)

As part of a Grafana training or observability training, master the declarative syntax of Grafana...

Grafana Alloy: collecting and transforming telemetry

Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...

Grafana Alloy best practices: optimizing your telemetry collection

Discover the golden rules for configuring Grafana Alloy robustly: modularity, secrets management,...

Grafana Alloy: Understanding and exploiting the User Interface (UI)

Discover how to enable, secure, and use Grafana Alloy's built-in web interface to visualize your ...

The LGTM stack (Loki, Grafana, Tempo, Mimir): architecture and setup

Grafana Labs' LGTM stack: Loki for logs, Grafana for visualization, Tempo for traces, Mimir for m...

Grafana Alloy: Log Management with Loki

Discover how to configure Grafana Alloy to read log files, journald, or network streams, process ...

OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines

Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...

Grafana Alloy components reference

A complete reference guide to all the components available in Grafana Alloy, organized by categor...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!