Lab: Writing, testing and measuring recording rules

From expensive queries to recorded series #

Context. Your dashboard recomputes the same aggregates on every refresh. You will record them in dedicated series, using Prometheus's own metrics, with no other dependency.

Prerequisites: Docker Compose, curl and jq  ·  Estimated time: 40 min

Create a working folder for this lab.

1. 1. Start Prometheus

   Why? Prometheus scrapes itself, so you immediately get real HTTP metrics.

   Write docker-compose.yml and prometheus.yml. Enable hot reload and reference a rules.yml file.

2. 2. Record a first aggregate

   Why? Staged aggregation avoids re-reading raw series at every level.

   Create rules.yml with the rate per job and handler, then the rate per job reusing the previous rule. Also create an empty tests.yml (touch tests.yml), check, start and query the new series after a minute.

3. 3. Add a robust error ratio and a percentile

   Why? Without care, the ratio stays empty until an error exists.

   Add the 5xx error rate, the error ratio (which must be 0 with no errors) and the 95th percentile of the duration. Reload and check the health of the rules.

4. 4. Write the unit tests

   Why? A wrong rule gives a plausible but wrong dashboard: only a test reveals it.

   In tests.yml, cover two cases: 10% errors (ratio 0.1) and no errors (ratio 0). Run the tests, then remove the or ... * 0 part of the ratio, see the second test fail, and put it back.

5. 5. Measure the gain

   Why? An optimisation is measured.

   Compare the samples read by the original query and by the recorded series, then find the evaluation time of your rules with prometheus_rule_group_last_duration_seconds. You should see a single sample for the recorded series and several hundred for the original query (the value depends on your instance).

Difficulty level : ●●●○○ (3/5)