3. Forwarding to Loki #
Secure forwarding to Grafana Loki.
The loki.write component #
loki.write "central" {
external_labels = { cluster = "paris" }
endpoint {
url = "https://loki.example.com/loki/api/v1/push"
tenant_id = "equipe-paiement"
batch_size = "1MiB"
batch_wait = "1s"
max_backoff_retries = 10
max_backoff_period = "5m"
basic_auth {
username = sys.env("LOKI_USER")
password = sys.env("LOKI_PASSWORD")
}
}
}| Setting | Purpose | Default |
|---|---|---|
batch_size | Batch size | 1MiB |
batch_wait | Maximum wait before sending | 1s |
min_backoff_period | First delay before a retry | 500ms |
max_backoff_period | Maximum delay between retries | 5m |
max_backoff_retries | Number of retries; 0 to retry forever | 10 |
tenant_id | Tenant, in multi-tenant mode | – |
What happens during an outage? #
Alloy retries with a growing delay. Once retries run out, the batch is lost and loki_write_dropped_entries_total increases. For file logs, the position protects what has not been read yet; for the rest, plan for the risk.
# Lines lost after retries ran out (alert on this)
rate(loki_write_dropped_entries_total[5m]) > 0
# Lines sent, retries
rate(loki_write_sent_entries_total[5m])
rate(loki_write_batch_retries_total[5m])Several destinations #
As for metrics, forward_to accepts several receivers: one loki.write for current storage, another for audit, with longer retention on the server side. external_labels are added when sending: keep them for what identifies the site, such as cluster.
Checking arrival #
We checked sending directly in Loki:
curl -s http://localhost:3100/loki/api/v1/labels
curl -s -G http://localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={job="app"}' \
--data-urlencode "start=$(( $(date +%s)-600 ))000000000"Best practice: keep few labels, with few values.job,env,levelandservicecover most needs; the rest is searched in the text at query time.
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Alloy : recevoir de l'OTLP et l'envoyer vers Tempo et Mimir✓ solved · 1 reply(ies)
Ask a question about this lesson →See all discussions about this course
Course Glossary
Grafana Alloy
An open-source, vendor-neutral telemetry collector designed to replace Grafana Agent, fully compatible with OpenTelemetry (OTel).
Recommended Articles
Collecting logs into Loki: Alloy, Promtail, Docker and OpenTelemetry
Send logs to Loki: Grafana Alloy, Promtail, the Docker driver, OTLP, good labels, parsing and enr...
Grafana Alloy: collecting and transforming telemetry
Learn how to use Grafana Alloy to collect, transform and route logs, metrics and traces. Discover...
Grafana Alloy vs Grafana Agent vs Promtail: which one to choose?
Discover how telemetry collectors have evolved in the Grafana ecosystem. A detailed comparison of...
Grafana Alloy: collecting system metrics and local logs
Learn how to configure Grafana Alloy to monitor the server it runs on, with a focus on configurat...
Grafana Alloy components reference
A complete reference guide to all the components available in Grafana Alloy, organized by categor...
The LGTM stack (Loki, Grafana, Tempo, Mimir): architecture and setup
Grafana Labs' LGTM stack: Loki for logs, Grafana for visualization, Tempo for traces, Mimir for m...
Grafana Alloy: Trace Management with Tempo
Dive into distributed trace processing. Learn how to ingest OTLP, Jaeger, or Zipkin traces with G...
OpenTelemetry Collector, Grafana Alloy and Bindplane: comparing three approaches to telemetry pipelines
Detailed comparison of three ways to manage telemetry pipelines: the OpenTelemetry Collector in Y...
Grafana Alloy: The importance of Self-Monitoring
Discover why and how to configure Grafana Alloy so that it monitors itself, collecting its own lo...