Grafana Alloy: Syntax and Configuration (Alloy Language: River)

1. Configuration blocks #

Attributes, blocks and expressions in River language.

Understanding the syntax #

Your alloy config is written in a file (usually config.alloy) using a declarative syntax strongly inspired by Terraform. In this alloy configuration, a component is defined by a name and an identifier. Inside the block, you define attributes (e.g., url = "http://...") and potentially sub-blocks. Components export values that can be referenced by other components (like alloy prometheus or prometheus.exporter.self), creating the pipeline.

prometheus.scrape "default" {
  targets = [{"__address__" = "localhost:9090"}]
  forward_to = [prometheus.remote_write.mimir.receiver]
}

Comma management #

In River, comma management in arrays and definitions is very flexible. It is recommended to add a comma after the last element of a list or multi-line configuration (trailing comma). This makes it easier to add subsequent elements and significantly reduces the risk of conflicts during code merges (Git merges).

prometheus.exporter.unix "example" {
  enable_collectors = [
    "cpu",
    "meminfo",
    "diskstats",
    "textfile",
  ]

  textfile {
    directory = "/var/lib/node_exporter/textfile_collector"
  }
}

2. River vs HCL #

What the Alloy syntax (formerly River) borrows from HCL, what sets it apart and why it is evaluated continuously.

From River to the “Alloy syntax” #

The language was called River in Grafana Agent Flow. Since Grafana Alloy 1.0, Grafana simply calls it the Alloy configuration syntax and files use the .alloy extension. The name “River” is still common in older documentation, articles and searches: it is the same language.

Its syntax is deliberately close to Terraform's HCL (HashiCorp Configuration Language): an engineer who writes Terraform reads an Alloy configuration effortlessly. But the two languages serve different purposes. HCL describes a target state reached once; Alloy describes a telemetry pipeline that runs all the time.

What the Alloy syntax borrows from HCL #

In both languages, a reference creates a dependency: the component reading a value depends on the component exporting it. This is how the graph is built without stating any execution order.

What sets them apart #

Side-by-side example #

The same need, a value that depends on the region, written in both languages:

# Terraform (HCL): evaluated once, at plan / apply
variable "region" {
  default = "eu-west-3"
}

locals {
  bucket_name = "logs-${var.region}"   # interpolation ${...}
}

resource "aws_s3_bucket" "logs" {
  count  = 2                           # meta-argument
  bucket = "${local.bucket_name}-${count.index}"
}

// Alloy: continuously re-evaluated whenever an exported value changes
local.file "region" {
  filename = "/etc/alloy/region"
}

prometheus.remote_write "mimir" {
  endpoint {
    // no interpolation: string.format() or the + operator
    url = string.format("https://mimir-%s.example.com/api/v1/push", local.file.region.content)
  }
  external_labels = {
    region = local.file.region.content,
    host   = constants.hostname,
  }
}

With Terraform, a region change means running terraform apply again. With Alloy, you just edit /etc/alloy/region: local.file re-reads it, exports the new value, and prometheus.remote_write reconfigures itself.

Continuous evaluation: the component graph #

At startup, Alloy builds a directed acyclic graph (DAG) from the references between components. Whenever a component exports a new value, only its dependants are re-evaluated. This is essential for observability, where targets appear and disappear all the time (Kubernetes pods, cloud instances).

// 1. Discovery: the pod list changes all the time
discovery.kubernetes "pods" {
  role = "pod"
}

// 2. Filtering: keep only annotated pods
discovery.relabel "annotated" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_pod_annotation_prometheus_io_scrape"]
    regex         = "true"
    action        = "keep"
  }
}

// 3. Scraping: its targets automatically follow steps 1 and 2
prometheus.scrape "pods" {
  targets    = discovery.relabel.annotated.output
  forward_to = [prometheus.remote_write.mimir.receiver]
}

When a pod is created, discovery.kubernetes exports a new list; discovery.relabel is re-evaluated, then prometheus.scrape receives its new targets. No restart, no manual action. The Alloy UI (port 12345) shows this graph and the health of every component.

Common pitfall: a circular reference (A reads B which reads A) is rejected at load time. The error message names the components involved; split the processing into two separate components.

Tools and commands to know #

# Check and format the syntax (like terraform fmt)
alloy fmt config.alloy
alloy fmt --write config.alloy

# Run Alloy with the debugging UI (component graph)
alloy run config.alloy --server.http.listen-addr=0.0.0.0:12345

# Reload the configuration without restarting
curl -X POST http://localhost:12345/-/reload

# Convert an existing configuration to the Alloy syntax
alloy convert --source-format=prometheus --output=config.alloy prometheus.yml
alloy convert --source-format=promtail   --output=logs.alloy   promtail.yml

Migration: alloy convert also accepts --source-format=static (Grafana Agent Static) and --source-format=otelcol (OpenTelemetry Collector). Always review the result: comments and some uncommon options are not carried over.

3. Variables and Functions #

Types, value sources, built-in functions and five concrete examples to make an Alloy configuration dynamic and reusable.

Value types #

An Alloy configuration handles typed values: numbers, strings, booleans, null, arrays [ ] and objects { }. Alloy adds two types of its own: secret, for passwords and tokens, and capsule, for internal values such as receivers or targets.

// Numbers, strings, booleans, null
scrape_interval = "30s"
sample_limit    = 5000
enabled         = true

// Array: trailing comma recommended
enable_collectors = ["cpu", "meminfo", "filesystem",]

// Object: identifier or string keys
targets = [
  { "__address__" = "app-1:8080", env = "prod" },
  { "__address__" = "app-2:8080", env = "prod" },
]

// Reference to a value exported by another component
forward_to = [prometheus.remote_write.mimir.receiver]

Type errors are caught at load time, for example a string passed where a number is expected. The message gives the file, line and faulty attribute.

Where do “variables” come from? #

There is no variable or locals block as in Terraform. A reusable value always comes from one of these sources:

The local.* and remote.* sources are components themselves. When their content changes, every component reading them is re-evaluated automatically.

For details on what Alloy reads from the environment, including OTEL_RESOURCE_ATTRIBUTES, see the Environment variables and OpenTelemetry lesson.

Operators and built-in functions #

Expressions accept arithmetic (+ - * / % ^), comparison (== != < <= > >=) and logical (&& || !) operators. The + operator also concatenates strings. Standard library functions are grouped by namespace:

Old names: in Grafana Agent Flow (River), these functions were called env(), format(), join(), concat(), json_decode() or nonsensitive(). If an example found online fails with “unknown function”, rename them with their namespace.

Example 1: one configuration per environment #

The same config.alloy file is deployed everywhere; environment-specific values are passed as environment variables (Docker, Helm, systemd).

// One file for staging and production:
// only the environment variables change.
prometheus.remote_write "mimir" {
  endpoint {
    // coalesce: first non-empty value (default when the variable is missing)
    url = coalesce(sys.env("MIMIR_URL"), "http://mimir:9009/api/v1/push")

    headers = {
      "X-Scope-OrgID" = coalesce(sys.env("MIMIR_TENANT"), "anonymous"),
    }
  }

  external_labels = {
    cluster     = coalesce(sys.env("CLUSTER_NAME"), "local"),
    environment = string.to_lower(coalesce(sys.env("ENV"), "DEV")),
    host        = constants.hostname,
  }
}

# Staging
MIMIR_URL=https://mimir-staging.example.com/api/v1/push CLUSTER_NAME=staging ENV=STAGING alloy run config.alloy

# Docker Compose
services:
  alloy:
    image: grafana/alloy:latest
    environment:
      MIMIR_URL: https://mimir.example.com/api/v1/push
      CLUSTER_NAME: prod-eu
      ENV: PROD

Example 2: a secret read from a file #

// The password is read from a mounted file (Kubernetes secret, Vault agent…)
local.file "mimir_password" {
  filename  = "/var/run/secrets/mimir/password"
  is_secret = true              // the value has the secret type: never shown in the UI
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "https://mimir.example.com/api/v1/push"
    basic_auth {
      username = sys.env("MIMIR_USER")
      password = local.file.mimir_password.content
    }
  }
}

With is_secret = true, the exported value has the secret type: it never shows up in the Alloy UI or in logs. If the file is updated (password rotation), Alloy re-reads it and reconfigures the output without a restart.

Never write a plain-text password in config.alloy: the file almost always ends up in a Git repository. Use local.file, remote.kubernetes.secret or remote.vault instead.

Example 3: targets described in a JSON file #

// targets.json, maintained by another tool (CMDB, script, Ansible…):
// [{"__address__": "db-1:9187", "role": "primary"}, {"__address__": "db-2:9187", "role": "replica"}]
local.file "targets" {
  filename = "/etc/alloy/targets.json"
}

prometheus.scrape "postgres" {
  // encoding.from_json turns the file content into an array of objects
  targets    = encoding.from_json(local.file.targets.content)
  forward_to = [prometheus.remote_write.mimir.receiver]
}
// The file is re-read automatically when it changes: new targets are scraped without a restart.

This pattern lets a team declare its targets in a simple file without touching the Alloy configuration. Fields other than __address__ (here role) become labels on the collected metrics.

Example 4: building strings and labels #

loki.process "normalize" {
  forward_to = [loki.write.default.receiver]

  stage.static_labels {
    values = {
      // Concatenation with +
      service = "checkout-" + coalesce(sys.env("REGION"), "eu"),
      // Formatting
      team    = string.to_upper("payments"),
      // Joining a list
      zones   = string.join(["eu-west-3a", "eu-west-3b"], ","),
    }
  }
}

There is no ${...} interpolation as in HCL: + concatenation is enough for simple cases, and string.format reads better as soon as several values are involved.

Example 5: factoring with declare and import #

To avoid copy-pasting the same block, Alloy offers custom components with declare. They are the equivalent of Terraform modules: input parameters (argument), internal components and exported values (export).

// Reusable custom component: the equivalent of a Terraform module
declare "mimir_writer" {
  argument "url" {}
  argument "tenant" {
    optional = true
    default  = "anonymous"
  }

  prometheus.remote_write "default" {
    endpoint {
      url     = argument.url.value
      headers = { "X-Scope-OrgID" = argument.tenant.value }
    }
  }

  export "receiver" {
    value = prometheus.remote_write.default.receiver
  }
}

// Two instances, two tenants, without duplicating the configuration
mimir_writer "prod" {
  url    = "https://mimir.example.com/api/v1/push"
  tenant = "prod"
}
mimir_writer "audit" {
  url    = "https://mimir.example.com/api/v1/push"
  tenant = "audit"
}

prometheus.scrape "app" {
  targets    = [{ "__address__" = "app:8080" }]
  forward_to = [mimir_writer.prod.receiver, mimir_writer.audit.receiver]
}

To share these components across teams or servers, put them in a separate file and load them with import.file, import.http or import.git:

// Load shared components from a Git repository (re-read every 5 minutes)
import.git "shared" {
  repository     = "https://github.com/acme/alloy-modules.git"
  revision       = "main"
  path           = "modules/mimir.alloy"
  pull_frequency = "5m"
}

shared.mimir_writer "prod" {
  url = sys.env("MIMIR_URL")
}

Best practices: one file per role (collection, processing, output); defaults with coalesce; secrets in local.file or a vault; alloy fmt in CI; and a look at the UI graph after every change.

4. Secrets Management #

Environment variables, files, Kubernetes Secrets and HashiCorp Vault: protecting and rotating Grafana Alloy secrets.

Why secrets need special care #

A telemetry collector handles many credentials: Grafana Cloud API keys, Mimir or Loki passwords, Datadog or Dynatrace tokens, TLS certificates, credentials for monitored databases. The config.alloy file, on the other hand, is meant to be versioned, reviewed and shared. A plain-text secret in that file ends up in Git, backups, screenshots and support tickets.

Grafana Alloy addresses this in three ways: a dedicated value type (secret), secret sources (environment variables, files, Kubernetes, Vault) and automatic re-evaluation when a secret changes. The session video introduces the approach; this article details it with examples.

// ❌ Never do this: the token ends up in Git, in backups and in tickets
prometheus.remote_write "cloud" {
  endpoint {
    url = "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
    basic_auth {
      username = "123456"
      password = "glc_eyJvIjoiMTIzNDU2Iiwibi..."
    }
  }
}

The secret type: what Alloy hides, and what it does not #

Sensitive component attributes, such as password, bearer_token or api_key, have the secret type. A value of this type is never displayed: the Alloy UI (port 12345) and the logs show (secret) instead.

Remember: masking protects display, not storage. The secret stays readable where it is stored (environment variable, file, Kubernetes Secret): that is where access must be restricted.

Method 1: environment variables #

The simplest and most portable method: the configuration reads the secret with sys.env(), and the deployment tool injects it.

// Environment variables: simple and universal (Docker, systemd, Kubernetes)
prometheus.remote_write "cloud" {
  endpoint {
    url = sys.env("GRAFANA_CLOUD_PROM_URL")
    basic_auth {
      username = sys.env("GRAFANA_CLOUD_PROM_USER")
      password = sys.env("GRAFANA_CLOUD_API_KEY")   // converted to a secret by the password attribute
    }
  }
}

# systemd: /etc/default/alloy (mode 600, owned by root)
GRAFANA_CLOUD_PROM_URL=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
GRAFANA_CLOUD_PROM_USER=123456
GRAFANA_CLOUD_API_KEY=glc_xxxxxxxx

# Docker: unversioned .env file (listed in .gitignore)
docker run --env-file ./alloy.env -v ./config.alloy:/etc/alloy/config.alloy grafana/alloy:latest \
  run /etc/alloy/config.alloy

Pros: nothing to install, works everywhere. Cons: the value is visible to anyone able to inspect the process (docker inspect, /proc/<pid>/environ), and a change requires restarting Alloy.

Method 2: a secret file read by local.file #

The secret is placed in a file with restricted permissions by Docker, Kubernetes, Ansible or a Vault agent. Alloy reads it with local.file and watches it: if the file changes, for example during a key rotation, the components using it are reconfigured without a restart.

// Mounted file: the secret goes neither through the environment nor through the configuration
local.file "api_key" {
  filename  = "/run/secrets/grafana_cloud_api_key"
  is_secret = true              // secret-typed value: hidden in the UI and logs
}

loki.write "cloud" {
  endpoint {
    url = sys.env("GRAFANA_CLOUD_LOKI_URL")
    basic_auth {
      username = sys.env("GRAFANA_CLOUD_LOKI_USER")
      password = local.file.api_key.content
    }
  }
}

# docker-compose.yml: Docker secret mounted in /run/secrets
services:
  alloy:
    image: grafana/alloy:latest
    command: run /etc/alloy/config.alloy --server.http.listen-addr=0.0.0.0:12345
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
    secrets:
      - grafana_cloud_api_key
secrets:
  grafana_cloud_api_key:
    file: ./secrets/grafana_cloud_api_key.txt   # mode 600, outside the Git repository

Zero-downtime rotation: replace the file content, Alloy detects the change and reopens connections with the new key. Keep the old key valid on the server side for a few minutes during the switch.

Method 3: Kubernetes Secrets #

On Kubernetes, two approaches coexist. The first mounts the Secret as an environment variable or a file through the Helm chart, then falls back to methods 1 and 2. The second uses remote.kubernetes.secret, which reads the Secret directly from the Kubernetes API and follows its changes.

# values.yaml of the grafana/alloy Helm chart: secret exposed as an environment variable
alloy:
  extraEnv:
    - name: GRAFANA_CLOUD_API_KEY
      valueFrom:
        secretKeyRef:
          name: grafana-cloud
          key: api-key

// Read a Kubernetes Secret directly (Alloy's service account must be allowed to read it)
remote.kubernetes.secret "mimir" {
  namespace = "monitoring"
  name      = "mimir-credentials"
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "https://mimir.example.com/api/v1/push"
    basic_auth {
      username = convert.nonsensitive(remote.kubernetes.secret.mimir.data["username"])
      password = remote.kubernetes.secret.mimir.data["password"]
    }
  }
}

# Least privilege: read ONE specific secret in ONE namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: alloy-read-mimir-credentials
  namespace: monitoring
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["mimir-credentials"]
    verbs: ["get", "watch", "list"]

With remote.kubernetes.secret there is no volume and no pod restart: a Secret update is applied within seconds. In return, Alloy's service account must be allowed to read that Secret; restrict it with resourceNames.

Method 4: HashiCorp Vault #

For organisations that centralise secrets in Vault, the remote.vault component reads a path from the vault and re-reads it periodically. It supports several authentication methods: auth.token, auth.approle, auth.kubernetes, auth.aws, auth.gcp, auth.userpass…

// HashiCorp Vault: Kubernetes authentication, re-read every 5 minutes
remote.vault "mimir" {
  server = "https://vault.example.com:8200"
  path   = "secret/data/monitoring/mimir"     // KV v2 engine

  auth.kubernetes {
    role = "alloy"
  }

  reread_frequency = "5m"
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "https://mimir.example.com/api/v1/push"
    bearer_token = remote.vault.mimir.data["token"]
  }
}

The Vault authentication token is itself a secret. Prefer authentication without a static secret (auth.kubernetes, auth.aws); otherwise read the token with local.file and is_secret = true.

Which method should you choose? #

Common mistakes and best practices #

• Plain-text secret in config.alloy: add a CI check (gitleaks, git grep) and revoke any key published by mistake immediately.
• convert.nonsensitive() used to silence an error: the secret becomes visible in the UI. Keep this function for truly public values such as a user name.
• World-readable secret file: mode 600, owned by the alloy user.
• Alloy UI exposed to the Internet: secrets are masked, but the topology and backend URLs remain visible. Listen on 127.0.0.1 or protect port 12345.
• One secret shared by every environment: use one key per environment and per purpose (metrics write, logs write), with the least privilege.

# Create the Kubernetes secret without writing it to a versioned file
kubectl -n monitoring create secret generic mimir-credentials \
  --from-literal=username=alloy --from-literal=password="$(openssl rand -base64 24)"

# Protect a secret file on a VM
sudo install -m 600 -o alloy -g alloy /dev/stdin /etc/alloy/secrets/api_key <<< "glc_xxxxxxxx"

# Check that no secret is versioned
git grep -nE "glc_|password *= *\"" -- '*.alloy'

# Check in the UI that the value is shown as (secret)
curl -s http://localhost:12345/api/v0/web/components | grep -c secret

5. Alloy Observability #

Monitor the health of your own pipelines via the native UI interface.

The local user interface #

Alloy embeds an HTTP server (via alloy port 12345 by default) that exposes the grafana alloy ui, a very useful graphical interface. This UI allows you to visualize the graph (DAG) of your components, check their state (Healthy/Unhealthy), and inspect the evaluated arguments and exports of each component in real-time. It is the essential tool for alloy live debugging of your configurations.