Writing and naming recording rules

3. Writing your first rules #

The syntax, a full example and group options.

A rules file #

Declare the file in prometheus.yml, then write the rules. These examples use Prometheus's own metrics, so you can try them straight away:

global:
  scrape_interval: 5s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

groups:
  - name: http_prometheus
    rules:
      - record: job_handler:prometheus_http_requests:rate5m
        expr: sum by (job, handler) (rate(prometheus_http_requests_total[5m]))

      - record: job:prometheus_http_requests:rate5m
        expr: sum by (job) (job_handler:prometheus_http_requests:rate5m)

      - record: job:prometheus_http_requests_errors:rate5m
        expr: sum by (job) (rate(prometheus_http_requests_total{code=~"5.."}[5m]))

      - record: job:prometheus_http_requests_errors:ratio_rate5m
        expr: |
          (job:prometheus_http_requests_errors:rate5m or job:prometheus_http_requests:rate5m * 0)
            / job:prometheus_http_requests:rate5m

      - record: job:prometheus_http_request_duration_seconds:p95_rate5m
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))

In Grafana, the query becomes very simple:

# Before: recomputed on every refresh of every panel
sum by (job) (rate(http_requests_total[5m]))

# After: simple read of the recorded series
job:http_requests:rate5m

Group options #

groups:
  - name: http_aggregates
    interval: 30s        # otherwise: the global evaluation_interval
    query_offset: 30s    # evaluates slightly in the past if data arrives late
    limit: 1000          # the rule fails if it produces more than 1000 series
    labels:
      team: platform     # added to all rules of the group
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

4. Naming your rules well #

The level:metric:operations convention.

The official convention #

The Prometheus documentation recommends the level:metric:operations format. The name then says what the series contains, without opening the rules file. Colons are reserved for this purpose: exporters must not use them.

5. Pitfalls to avoid #

Ratios, missing series, order of operations.

Never average ratios #

A service at 10 requests per second with 50% errors and another at 1000 requests per second with 0% do not average 25% errors, but about 0.5%. So record the numerator and denominator, then divide their sums:

# Wrong: average of ratios, a small service weighs as much as a big one
avg by (job) (instance:http_requests_errors:ratio_rate5m)

# Right: aggregate the numerator and denominator separately
sum by (job) (instance:http_requests_errors:rate5m)
  / sum by (job) (instance:http_requests:rate5m)

The ratio that disappears when all is well #

This is the sneakiest pitfall, observed while preparing this chapter. If no 5xx series exists, the filter returns nothing, and dividing by an empty result gives an empty result, not 0. The dashboard shows “No data”: you can then no longer tell “no errors” from a collection outage.

# Empty as long as no 5xx error has been seen
job:requests_errors:rate5m / job:requests:rate5m

# Equals 0 when there is no error
(job:requests_errors:rate5m or job:requests:rate5m * 0) / job:requests:rate5m

Other frequent mistakes #

• rate first, sum second: sum(rate(x[5m])). Adding counters before rate breaks the detection of their resets.
• Histograms: keep the le label in sum by, or histogram_quantile cannot compute anything.
• Too many labels: a rule keeping instance or pod keeps almost all the cardinality.
• Changing a rule: if the expression changes meaning, change the name too, or the old and new logic mix in the same series.