Lab: Writing, testing and measuring recording rules with correction

From expensive queries to recorded series #

Context. Your dashboard recomputes the same aggregates on every refresh. You will record them in dedicated series, using Prometheus's own metrics, with no other dependency.

Prerequisites: Docker Compose, curl and jq  ·  Estimated time: 40 min

Create a working folder for this lab.

1. 1. Start Prometheus

   Why? Prometheus scrapes itself, so you immediately get real HTTP metrics.

   Write docker-compose.yml and prometheus.yml. Enable hot reload and reference a rules.yml file.
   See the correction

   services:
     prometheus:
       image: prom/prometheus:latest
       command:
         - --config.file=/etc/prometheus/prometheus.yml
         - --web.enable-lifecycle
       ports:
         - "9090:9090"
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
         - ./rules.yml:/etc/prometheus/rules.yml:ro
         - ./tests.yml:/etc/prometheus/tests.yml:ro

   global:
     scrape_interval: 5s
     evaluation_interval: 15s
   
   rule_files:
     - /etc/prometheus/rules.yml
   
   scrape_configs:
     - job_name: prometheus
       static_configs:
         - targets: ["localhost:9090"]

2. 2. Record a first aggregate

   Why? Staged aggregation avoids re-reading raw series at every level.

   Create rules.yml with the rate per job and handler, then the rate per job reusing the previous rule. Also create an empty tests.yml (touch tests.yml), check, start and query the new series after a minute.
   See the correction

   groups:
     - name: http_prometheus
       rules:
         - record: job_handler:prometheus_http_requests:rate5m
           expr: sum by (job, handler) (rate(prometheus_http_requests_total[5m]))
   
         - record: job:prometheus_http_requests:rate5m
           expr: sum by (job) (job_handler:prometheus_http_requests:rate5m)

   docker compose up -d
   docker compose exec prometheus promtool check rules /etc/prometheus/rules.yml
   curl -X POST http://localhost:9090/-/reload
   curl -s --get --data-urlencode 'query=job:prometheus_http_requests:rate5m' http://localhost:9090/api/v1/query

3. 3. Add a robust error ratio and a percentile

   Why? Without care, the ratio stays empty until an error exists.

   Add the 5xx error rate, the error ratio (which must be 0 with no errors) and the 95th percentile of the duration. Reload and check the health of the rules.
   See the correction

   groups:
     - name: http_prometheus
       rules:
         - record: job_handler:prometheus_http_requests:rate5m
           expr: sum by (job, handler) (rate(prometheus_http_requests_total[5m]))
   
         - record: job:prometheus_http_requests:rate5m
           expr: sum by (job) (job_handler:prometheus_http_requests:rate5m)
   
         - record: job:prometheus_http_requests_errors:rate5m
           expr: sum by (job) (rate(prometheus_http_requests_total{code=~"5.."}[5m]))
   
         - record: job:prometheus_http_requests_errors:ratio_rate5m
           expr: |
             (job:prometheus_http_requests_errors:rate5m or job:prometheus_http_requests:rate5m * 0)
               / job:prometheus_http_requests:rate5m
   
         - record: job:prometheus_http_request_duration_seconds:p95_rate5m
           expr: |
             histogram_quantile(0.95,
               sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))

   curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | {name, health, evaluationTime}'

4. 4. Write the unit tests

   Why? A wrong rule gives a plausible but wrong dashboard: only a test reveals it.

   In tests.yml, cover two cases: 10% errors (ratio 0.1) and no errors (ratio 0). Run the tests, then remove the or ... * 0 part of the ratio, see the second test fail, and put it back.
   See the correction

   rule_files:
     - rules.yml
   evaluation_interval: 1m
   
   tests:
     - name: 10% errors
       interval: 1m
       input_series:
         - series: 'prometheus_http_requests_total{job="api",handler="/a",code="200"}'
           values: '0+54x10'
         - series: 'prometheus_http_requests_total{job="api",handler="/a",code="500"}'
           values: '0+6x10'
       promql_expr_test:
         - expr: job:prometheus_http_requests:rate5m
           eval_time: 10m
           exp_samples:
             - labels: 'job:prometheus_http_requests:rate5m{job="api"}'
               value: 1
         - expr: job:prometheus_http_requests_errors:ratio_rate5m
           eval_time: 10m
           exp_samples:
             - labels: 'job:prometheus_http_requests_errors:ratio_rate5m{job="api"}'
               value: 0.1
   
     - name: no errors, the ratio is 0 and not empty
       interval: 1m
       input_series:
         - series: 'prometheus_http_requests_total{job="api",handler="/a",code="200"}'
           values: '0+60x10'
       promql_expr_test:
         - expr: job:prometheus_http_requests_errors:ratio_rate5m
           eval_time: 10m
           exp_samples:
             - labels: 'job:prometheus_http_requests_errors:ratio_rate5m{job="api"}'
               value: 0

   docker compose exec prometheus promtool test rules /etc/prometheus/tests.yml

5. 5. Measure the gain

   Why? An optimisation is measured.

   Compare the samples read by the original query and by the recorded series, then find the evaluation time of your rules with prometheus_rule_group_last_duration_seconds. You should see a single sample for the recorded series and several hundred for the original query (the value depends on your instance).
   See the correction

   for q in 'sum by (job) (rate(prometheus_http_requests_total[5m]))' 'job:prometheus_http_requests:rate5m'; do
     curl -s --get --data-urlencode "query=$q" --data-urlencode 'stats=all' \
       http://localhost:9090/api/v1/query | jq .data.stats.samples.totalQueryableSamples
   done

Difficulty level : ●●●○○ (3/5)