Understanding recording rules

1. Why recording rules? #

Compute in advance rather than on every read.

The problem #

A PromQL query is computed when it runs. On a twenty-panel dashboard, refreshed every thirty seconds and opened by ten people, Prometheus keeps repeating the same computations on the same samples. The dashboard gets slow, and every extra user makes it worse.

A recording rule solves this: Prometheus evaluates the expression at a regular interval and writes the result to a new series. Reads become simple accesses to that series.

Real measurement: on a test Prometheus 3.15, with stats=all, sum by (job) (rate(prometheus_http_requests_total[5m])) reads 448 samples, and the equivalent recorded series just one. The gap is far larger on a production server.

When to use them, and when not to #

A rule is not free: each recorded series uses storage and memory, like a scraped series. The gain comes from aggregation: replacing thousands of series with a few dozen.

2. How Prometheus evaluates them #

Groups, interval, order and no retroactive effect.

How it works #

• Rules live in groups, declared in files referenced by rule_files.
• Within a group they run in order, at the same timestamp. A rule can therefore use the result of the previous one.
• The interval is the group's (interval), otherwise the global evaluation_interval.
• The result keeps the labels of the expression; the metric name is replaced by record.
• A rule only computes the present: a new rule has no history, unless recomputed manually (see below).

Mind the interval: a series with no point for more than five minutes (the default lookback) no longer appears in instant queries. A group evaluated every ten minutes therefore produces gaps.