Integrate environment variables and vault files.
4. Secrets Management #
Environment variables, files, Kubernetes Secrets and HashiCorp Vault: protecting and rotating Grafana Alloy secrets.
Why secrets need special care #
A telemetry collector handles many credentials: Grafana Cloud API keys, Mimir or Loki passwords, Datadog or Dynatrace tokens, TLS certificates, credentials for monitored databases. The config.alloy file, on the other hand, is meant to be versioned, reviewed and shared. A plain-text secret in that file ends up in Git, backups, screenshots and support tickets.
Grafana Alloy addresses this in three ways: a dedicated value type (secret), secret sources (environment variables, files, Kubernetes, Vault) and automatic re-evaluation when a secret changes. The session video introduces the approach; this article details it with examples.
// ❌ Never do this: the token ends up in Git, in backups and in tickets
prometheus.remote_write "cloud" {
endpoint {
url = "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
basic_auth {
username = "123456"
password = "glc_eyJvIjoiMTIzNDU2Iiwibi..."
}
}
}The secret type: what Alloy hides, and what it does not #
Sensitive component attributes, such as password, bearer_token or api_key, have the secret type. A value of this type is never displayed: the Alloy UI (port 12345) and the logs show (secret) instead.
| Situation | Behaviour |
|---|---|
A string is assigned to a secret attribute | It is automatically converted to a secret |
A secret is assigned to a string attribute (e.g. username, a label) | Load-time error: the conversion must be explicit |
Explicit conversion with convert.nonsensitive(...) | The value becomes a visible string again: keep it for non-sensitive data (a user name) |
local.file with is_secret = true | The exported content is a secret |
remote.kubernetes.secret and remote.vault | Every value exported in data is a secret |
Remember: masking protects display, not storage. The secret stays readable where it is stored (environment variable, file, Kubernetes Secret): that is where access must be restricted.
Method 1: environment variables #
The simplest and most portable method: the configuration reads the secret with sys.env(), and the deployment tool injects it.
// Environment variables: simple and universal (Docker, systemd, Kubernetes)
prometheus.remote_write "cloud" {
endpoint {
url = sys.env("GRAFANA_CLOUD_PROM_URL")
basic_auth {
username = sys.env("GRAFANA_CLOUD_PROM_USER")
password = sys.env("GRAFANA_CLOUD_API_KEY") // converted to a secret by the password attribute
}
}
}# systemd: /etc/default/alloy (mode 600, owned by root)
GRAFANA_CLOUD_PROM_URL=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
GRAFANA_CLOUD_PROM_USER=123456
GRAFANA_CLOUD_API_KEY=glc_xxxxxxxx
# Docker: unversioned .env file (listed in .gitignore)
docker run --env-file ./alloy.env -v ./config.alloy:/etc/alloy/config.alloy grafana/alloy:latest \
run /etc/alloy/config.alloyPros: nothing to install, works everywhere. Cons: the value is visible to anyone able to inspect the process (docker inspect, /proc/<pid>/environ), and a change requires restarting Alloy.
Method 2: a secret file read by local.file #
The secret is placed in a file with restricted permissions by Docker, Kubernetes, Ansible or a Vault agent. Alloy reads it with local.file and watches it: if the file changes, for example during a key rotation, the components using it are reconfigured without a restart.
// Mounted file: the secret goes neither through the environment nor through the configuration
local.file "api_key" {
filename = "/run/secrets/grafana_cloud_api_key"
is_secret = true // secret-typed value: hidden in the UI and logs
}
loki.write "cloud" {
endpoint {
url = sys.env("GRAFANA_CLOUD_LOKI_URL")
basic_auth {
username = sys.env("GRAFANA_CLOUD_LOKI_USER")
password = local.file.api_key.content
}
}
}# docker-compose.yml: Docker secret mounted in /run/secrets
services:
alloy:
image: grafana/alloy:latest
command: run /etc/alloy/config.alloy --server.http.listen-addr=0.0.0.0:12345
volumes:
- ./config.alloy:/etc/alloy/config.alloy:ro
secrets:
- grafana_cloud_api_key
secrets:
grafana_cloud_api_key:
file: ./secrets/grafana_cloud_api_key.txt # mode 600, outside the Git repositoryZero-downtime rotation: replace the file content, Alloy detects the change and reopens connections with the new key. Keep the old key valid on the server side for a few minutes during the switch.
Method 3: Kubernetes Secrets #
On Kubernetes, two approaches coexist. The first mounts the Secret as an environment variable or a file through the Helm chart, then falls back to methods 1 and 2. The second uses remote.kubernetes.secret, which reads the Secret directly from the Kubernetes API and follows its changes.
# values.yaml of the grafana/alloy Helm chart: secret exposed as an environment variable
alloy:
extraEnv:
- name: GRAFANA_CLOUD_API_KEY
valueFrom:
secretKeyRef:
name: grafana-cloud
key: api-key// Read a Kubernetes Secret directly (Alloy's service account must be allowed to read it)
remote.kubernetes.secret "mimir" {
namespace = "monitoring"
name = "mimir-credentials"
}
prometheus.remote_write "mimir" {
endpoint {
url = "https://mimir.example.com/api/v1/push"
basic_auth {
username = convert.nonsensitive(remote.kubernetes.secret.mimir.data["username"])
password = remote.kubernetes.secret.mimir.data["password"]
}
}
}# Least privilege: read ONE specific secret in ONE namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: alloy-read-mimir-credentials
namespace: monitoring
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["mimir-credentials"]
verbs: ["get", "watch", "list"]With remote.kubernetes.secret there is no volume and no pod restart: a Secret update is applied within seconds. In return, Alloy's service account must be allowed to read that Secret; restrict it with resourceNames.
Method 4: HashiCorp Vault #
For organisations that centralise secrets in Vault, the remote.vault component reads a path from the vault and re-reads it periodically. It supports several authentication methods: auth.token, auth.approle, auth.kubernetes, auth.aws, auth.gcp, auth.userpass…
// HashiCorp Vault: Kubernetes authentication, re-read every 5 minutes
remote.vault "mimir" {
server = "https://vault.example.com:8200"
path = "secret/data/monitoring/mimir" // KV v2 engine
auth.kubernetes {
role = "alloy"
}
reread_frequency = "5m"
}
prometheus.remote_write "mimir" {
endpoint {
url = "https://mimir.example.com/api/v1/push"
bearer_token = remote.vault.mimir.data["token"]
}
}The Vault authentication token is itself a secret. Prefer authentication without a static secret (auth.kubernetes,auth.aws); otherwise read the token withlocal.fileandis_secret = true.
Which method should you choose? #
| Method | Rotation without restart | Complexity | Use case |
|---|---|---|---|
sys.env() | No | Very low | Workstation, simple VM, Docker |
local.file | Yes | Low | Docker secrets, files placed by Ansible or a Vault agent |
remote.kubernetes.secret | Yes | Medium (RBAC) | Kubernetes clusters |
remote.vault | Yes (reread_frequency) | Higher | Centralised secrets, automatic rotation, auditing |
Common mistakes and best practices #
- Plain-text secret in
config.alloy: add a CI check (gitleaks,git grep) and revoke any key published by mistake immediately. convert.nonsensitive()used to silence an error: the secret becomes visible in the UI. Keep this function for truly public values such as a user name.- World-readable secret file: mode
600, owned by thealloyuser. - Alloy UI exposed to the Internet: secrets are masked, but the topology and backend URLs remain visible. Listen on
127.0.0.1or protect port 12345. - One secret shared by every environment: use one key per environment and per purpose (metrics write, logs write), with the least privilege.
# Create the Kubernetes secret without writing it to a versioned file
kubectl -n monitoring create secret generic mimir-credentials \
--from-literal=username=alloy --from-literal=password="$(openssl rand -base64 24)"
# Protect a secret file on a VM
sudo install -m 600 -o alloy -g alloy /dev/stdin /etc/alloy/secrets/api_key <<< "glc_xxxxxxxx"
# Check that no secret is versioned
git grep -nE "glc_|password *= *\"" -- '*.alloy'
# Check in the UI that the value is shown as (secret)
curl -s http://localhost:12345/api/v0/web/components | grep -c secretMy progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Alloy : recevoir de l'OTLP et l'envoyer vers Tempo et Mimir✓ solved · 1 reply(ies)
Ask a question about this lesson →See all discussions about this course
Course Glossary
Grafana Alloy
An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).
Recommended Articles
Grafana Alloy best practices: optimizing your telemetry collection
Discover the golden rules for configuring Grafana Alloy robustly: modularity, secrets management,...
Grafana Alloy environment variables: GOMEMLIMIT, sys.env and OTEL_RESOURCE_ATTRIBUTES
Which variables does Grafana Alloy actually read? GOMEMLIMIT, HTTP_PROXY, sys.env, OTEL_RESOURCE_...
Grafana Alloy: The importance of Self-Monitoring
Discover why and how to configure Grafana Alloy so that it monitors itself, collecting its own lo...
Grafana Alloy: collecting system metrics and local logs
Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...
OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines
Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...
Grafana Alloy: collecting and transforming telemetry
Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...
Grafana Alloy: Log Management with Loki
Discover how to configure Grafana Alloy to read log files, journald, or network streams, process ...
Grafana Alloy components reference
A complete reference guide to all the components available in Grafana Alloy, organized by categor...
Alloy and the Grafana Labs ecosystem: which component for which tool?
The map of Alloy integrations with Faro, Beyla, k6, Loki, Mimir, Tempo, Pyroscope, Database Obser...