1. Clustering mode
Ensuring high availability and load balancing.
Resilience and Horizontal Scaling
In vast environments, a single instance of Alloy is not enough to collect from tens of thousands of targets. Alloy's Clustering mode allows launching multiple instances that communicate with each other (via a Gossip protocol). Together, they form a cluster that intelligently distributes the workload (sharding Prometheus scraping targets or log files). If a cluster node fails, the other nodes immediately take over its load, ensuring highly available telemetry collection without data duplication.
Best Practice: In Kubernetes, always deploy Alloy in Cluster mode as aStatefulSetrather than aDeployment. The StatefulSet guarantees stable network identities, which are essential for proper load balancing (Gossip) between nodes.
Common Mistake: Forgetting to configure persistent storage (PVC) on each cluster node for the WAL. If a pod restarts or is rescheduled, unsent metrics will be permanently lost.
# Helm configuration snippet (values.yaml)
alloy:
clustering:
enabled: true
name: "my-alloy-cluster"
controller:
type: statefulset
replicas: 3
volumeClaimTemplates:
- metadata:
name: alloy-wal
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
2. Alloy on Kubernetes
Using the Helm Chart and managing ConfigMaps.
Kubernetes deployment strategies
Using Grafana Alloy on Kubernetes is made easy by the official Helm Chart. Depending on your needs, the topology varies:
- DaemonSet: Ideal for collecting system logs (host, containers) and node metrics. An instance of Alloy runs on each node in the cluster.
- Deployment / StatefulSet (Cluster): Recommended for scraping external services or receiving pushed telemetry via OTLP. Coupled with clustering mode, this configuration provides the necessary scalability to handle high ingestion volumes.
The Helm Chart also allows dynamically managing River configurations via ConfigMaps, facilitating rolling updates and integration into GitOps pipelines (like ArgoCD or Flux).