1. The principle #
Describe, generate, load.
Why Sloth #
Sloth is an open source generator: from an SLO specification, it produces the SLI recording rules for all windows, the metadata (objective, remaining budget) and the multi-window, multi-burn-rate alerts of the Google method.
2. Writing a specification #
The prometheus/v1 format.
Example #
version: "prometheus/v1"
service: "checkout"
labels:
owner: "equipe-paiement"
slos:
- name: "requests-availability"
objective: 99.9
description: "Requêtes de l'API checkout sans erreur 5xx."
sli:
events:
error_query: sum(rate(http_requests_total{service="checkout",code=~"5.."}[{{.window}}]))
total_query: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
alerting:
name: CheckoutHighErrorRate
labels:
category: availability
annotations:
summary: "Budget d'erreur de checkout consommé trop vite"
runbook_url: https://wiki.example.org/runbooks/checkout
page_alert:
labels: { severity: critical }
ticket_alert:
labels: { severity: warning }The {{.window}} template is replaced by each window (5m, 30m, 1h, 2h, 6h, 1d, 3d, 30d).
sloth validate -i slos/
sloth generate -i slos/checkout.yml -o rules/checkout-slo.yml
# ou avec Docker
docker run --rm -v $PWD:/w ghcr.io/slok/sloth generate -i /w/slos/checkout.yml -o /w/rules/checkout-slo.ymlWhat is generated #
| Metric | Meaning |
|---|---|
slo:sli_error:ratio_rate5m ... ratio_rate30d | Error rate per window |
slo:objective:ratio | Objective (0.999) |
slo:error_budget:ratio | Budget (0.001) |
slo:current_burn_rate:ratio | Current burn rate |
slo:period_error_budget_remaining:ratio | Budget remaining over the period |
All carry the sloth_service, sloth_slo and sloth_id labels.
3. On Kubernetes and in Grafana #
Operator and dashboards.
Operator #
In Kubernetes mode, Sloth watches PrometheusServiceLevel resources and creates the corresponding PrometheusRule objects for the Prometheus Operator:
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
name: checkout
namespace: paiement
spec:
service: checkout
slos:
- name: requests-availability
objective: 99.9
sli:
events:
errorQuery: sum(rate(http_requests_total{service="checkout",code=~"5.."}[{{.window}}]))
totalQuery: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
alerting:
name: CheckoutHighErrorRate
pageAlert: { labels: { severity: critical } }
ticketAlert: { labels: { severity: warning } }Good practice: Sloth publishes two Grafana dashboards (IDs 14348 for the details of an SLO and 14643 for the overview) that directly use the generated metrics.
