Testing, deploying and monitoring rules

6. Testing rules with promtool #

Checks and unit tests, before deploying.

Unit tests, with no server #

promtool check rules checks the syntax. promtool test rules goes further: it injects fake series, evaluates the rules and compares the result. The test below covers both cases of the previous pitfall:

rule_files:
  - rules.yml
evaluation_interval: 1m

tests:
  - name: 10% errors
    interval: 1m
    input_series:
      - series: 'prometheus_http_requests_total{job="api",handler="/a",code="200"}'
        values: '0+54x10'
      - series: 'prometheus_http_requests_total{job="api",handler="/a",code="500"}'
        values: '0+6x10'
    promql_expr_test:
      - expr: job:prometheus_http_requests:rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:prometheus_http_requests:rate5m{job="api"}'
            value: 1
      - expr: job:prometheus_http_requests_errors:ratio_rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:prometheus_http_requests_errors:ratio_rate5m{job="api"}'
            value: 0.1

  - name: no errors, the ratio is 0 and not empty
    interval: 1m
    input_series:
      - series: 'prometheus_http_requests_total{job="api",handler="/a",code="200"}'
        values: '0+60x10'
    promql_expr_test:
      - expr: job:prometheus_http_requests_errors:ratio_rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:prometheus_http_requests_errors:ratio_rate5m{job="api"}'
            value: 0

How to read values: '0+54x10': the series starts at 0 and grows by 54 at each of the 10 one-minute steps, that is 0.9 requests per second. With 0.1 failing requests, the expected ratio is indeed 0.1.

7. Deploying #

Reloading the configuration, Kubernetes.

Reloading without a restart #

After a change, send SIGHUP to the process, or call POST /-/reload if Prometheus runs with --web.enable-lifecycle. An error in the file leaves the old configuration in place, so check the result in Status > Rule health or with /api/v1/rules.

--web.enable-lifecycle lets anyone reaching the port reload or stop Prometheus. Protect this access in production.

On Kubernetes #

With the Prometheus Operator, rules are declared by a PrometheusRule resource. Its label must match your Prometheus ruleSelector, or it is silently ignored.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: http-recording-rules
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match the ruleSelector of your Prometheus
spec:
  groups:
    - name: http
      rules:
        - record: job:http_requests:rate5m
          expr: sum by (job) (rate(http_requests_total[5m]))

8. Monitoring your rules #

Detecting a slow group or a failing rule.

Metrics to watch #

If a group takes longer to evaluate than its interval, evaluations are skipped and gaps appear. Prometheus exposes everything needed to detect it:

# Does an evaluation exceed its interval? (watch this first)
rate(prometheus_rule_group_iterations_missed_total[5m]) > 0

# Share of the interval used by each group
prometheus_rule_group_last_duration_seconds / prometheus_rule_group_interval_seconds

# Failing rules
rate(prometheus_rule_evaluation_failures_total[5m]) > 0

Best practice: alert on prometheus_rule_group_iterations_missed_total. If a group becomes too slow, split it: groups run in parallel.

9. Recomputing the past #

Backfilling with promtool.

Creating a history for a new rule #

promtool tsdb create-blocks-from rules queries a Prometheus API and writes TSDB blocks to the data/ folder (--output-dir option). They must then be moved to the data directory (storage.tsdb.path): they merge with existing blocks at the next compaction.

Documented limits: a rule depending on another rule being backfilled is not supported: backfill the base rules first, move their blocks, then run again for the dependent rules. Alerts in the file are ignored. Running again over an overlapping period creates blocks with the same data each time.

Commands to know #

# Check the syntax and count the rules
promtool check rules rules.yml

# Run the unit tests
promtool test rules tests.yml

# Reload without restarting (with --web.enable-lifecycle)
curl -X POST http://localhost:9090/-/reload

# List rules, their health and duration
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | {name, evaluationTime, rules: [.rules[] | {name, health}]}'

# Measure the cost of a query
curl -s --get --data-urlencode 'query=job:prometheus_http_requests:rate5m' --data-urlencode 'stats=all' \
  http://localhost:9090/api/v1/query | jq .data.stats.samples.totalQueryableSamples

# Recompute the past (creating TSDB blocks)
promtool tsdb create-blocks-from rules --start 2026-11-01T00:00:00Z --end 2026-11-02T00:00:00Z \
  --url http://localhost:9090 --eval-interval 60s rules.yml

Official sources #

• Prometheus : defining recording rules
• Prometheus : recording rules best practices
• Prometheus : unit testing for rules
• Prometheus : storage et backfilling