RousselTMLEARNING

Observability

Generating Prometheus SLOs with Sloth

Describe an SLO in a few lines with Sloth and automatically generate multi-window recording rules and burn rate alerts, via CLI or on Kubernetes, with its Grafana dashboards.

1. The principle #

Describe, generate, load.

Why Sloth #

How Sloth works
About twenty lines of specification produce dozens of consistent rules.

Sloth is an open source generator: from an SLO specification, it produces the SLI recording rules for all windows, the metadata (objective, remaining budget) and the multi-window, multi-burn-rate alerts of the Google method.

2. Writing a specification #

The prometheus/v1 format.

Example #

version: "prometheus/v1"
service: "checkout"
labels:
  owner: "equipe-paiement"
slos:
  - name: "requests-availability"
    objective: 99.9
    description: "Requêtes de l'API checkout sans erreur 5xx."
    sli:
      events:
        error_query: sum(rate(http_requests_total{service="checkout",code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
    alerting:
      name: CheckoutHighErrorRate
      labels:
        category: availability
      annotations:
        summary: "Budget d'erreur de checkout consommé trop vite"
        runbook_url: https://wiki.example.org/runbooks/checkout
      page_alert:
        labels: { severity: critical }
      ticket_alert:
        labels: { severity: warning }

The {{.window}} template is replaced by each window (5m, 30m, 1h, 2h, 6h, 1d, 3d, 30d).

sloth validate -i slos/
sloth generate -i slos/checkout.yml -o rules/checkout-slo.yml
# ou avec Docker
docker run --rm -v $PWD:/w ghcr.io/slok/sloth generate -i /w/slos/checkout.yml -o /w/rules/checkout-slo.yml

What is generated #

MetricMeaning
slo:sli_error:ratio_rate5m ... ratio_rate30dError rate per window
slo:objective:ratioObjective (0.999)
slo:error_budget:ratioBudget (0.001)
slo:current_burn_rate:ratioCurrent burn rate
slo:period_error_budget_remaining:ratioBudget remaining over the period

All carry the sloth_service, sloth_slo and sloth_id labels.

3. On Kubernetes and in Grafana #

Operator and dashboards.

Operator #

In Kubernetes mode, Sloth watches PrometheusServiceLevel resources and creates the corresponding PrometheusRule objects for the Prometheus Operator:

apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: checkout
  namespace: paiement
spec:
  service: checkout
  slos:
    - name: requests-availability
      objective: 99.9
      sli:
        events:
          errorQuery: sum(rate(http_requests_total{service="checkout",code=~"5.."}[{{.window}}]))
          totalQuery: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
      alerting:
        name: CheckoutHighErrorRate
        pageAlert: { labels: { severity: critical } }
        ticketAlert: { labels: { severity: warning } }
Good practice: Sloth publishes two Grafana dashboards (IDs 14348 for the details of an SLO and 14643 for the overview) that directly use the generated metrics.

Related articles

Time-based vs event-based SLO: understanding, computing and choosing (with the Qonto example)

Differences between time-based SLOs (time slices, uptime) and event-based SLOs (good events over total): defi…

Chaos engineering: principles, tools and practice with SLOs

A complete guide to chaos engineering: principles, steady state and hypothesis, blast radius, link with SLOs …

SRE: SLI, SLO and SLA explained simply

The fundamentals of Site Reliability Engineering (SRE): the difference between SLI, SLO and SLA, choosing ind…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!