CrowdSec observability: banned IPs, blocked attacks, Prometheus and Grafana

1. Answering the right questions #

The tools available.

Overview #

Prometheus, Grafana, Alertmanager, cscli, notifications and console let you follow CrowdSec

Question	Answer
Which IPs are banned?	`cscli decisions list`
Which attacks were detected?	`cscli alerts list --since 24h`
How many packets or requests were blocked?	`cscli metrics show bouncers`
Are my logs read and understood?	`cscli metrics show acquisition parsers`
Which scenarios trigger?	`cscli metrics show scenarios`
Are my bouncers alive?	`cscli bouncers list`

2. With cscli #

The immediate view.

Tracking banned IPs #

sudo cscli decisions list
sudo cscli decisions list --origin crowdsec
sudo cscli decisions list --origin CAPI
sudo cscli decisions list -o json

| ID | Source   | Scope:Value     | Reason               | Action | Events | expiration |
| 1  | crowdsec | Ip:203.0.113.50 | crowdsecurity/ssh-bf | ban    | 6      | 3h59m46s   |

The Source column shows the origin: crowdsec (local detection), cscli (manual), CAPI (community).

Analyzing an attack #

sudo cscli alerts list --since 24h
sudo cscli alerts inspect 3 --details

- ID           : 3
 - Machine      : test
 - Simulation   : false
 - Remediation  : true
 - Reason       : crowdsecurity/ssh-bf
 - Events Count : 6
 - Scope:Value  : Ip:203.0.113.50

Measuring actual blocking #

A decision does not say whether anything was blocked. Only the bouncer knows: cscli metrics show bouncers shows, per origin, the number of active addresses and the blocked volume (CrowdSec 1.6.3 and firewall bouncer 0.0.30 minimum).

sudo cscli metrics show bouncers
sudo nft list table ip crowdsec   # firewall counters

Troubleshooting #

Symptom	Likely cause	To check
No alerts	Undeclared log	`cscli metrics show acquisition`
Many Unparsed lines	Wrong `type`, missing collection	`cscli explain`
Decision, but no block	No bouncer, wrong IP seen	`cscli bouncers list`
Legitimate IP banned	False positive	`cscli alerts inspect`, allowlist

3. Prometheus and Grafana #

History and dashboards.

Exposing metrics #

In config.yaml:

prometheus:
  enabled: true
  level: full            # none, aggregated or full
  listen_addr: 0.0.0.0   # 127.0.0.1 by default: open it for a remote Prometheus
  listen_port: 6060

curl -s http://127.0.0.1:6060/metrics | grep -E '^cs_(active_decisions|alerts)'
# cs_active_decisions{action="ban",origin="crowdsec",reason="crowdsecurity/ssh-bf"} 1
# cs_alerts{kind="crowdsec",reason="crowdsecurity/ssh-bf"} 1

scrape_configs:
  - job_name: crowdsec
    static_configs:
      - targets: ["crowdsec:6060"]

Detail level. The aggregated level notably removes cs_active_decisions, cs_alerts and per-bouncer metrics. Keep full to track blocked IPs.

Useful metrics #

Metric	What it measures
`cs_active_decisions`	Active decisions by reason, origin and action
`cs_alerts`	Alerts by reason
`cs_bucket_overflowed_total`	Overflows per scenario: detected attacks
`cs_parser_hits_ok_total`, `cs_parser_hits_ko_total`	Lines understood or not
`cs_filesource_hits_total`	Lines read per file
`cs_lapi_bouncer_requests_total`	Activity of each bouncer
`cs_appsec_reqs_total`, `cs_appsec_block_total`	Requests inspected and blocked by AppSec
`cs_capi_last_pull_timestamp`	Last download of the community list

PromQL queries #

# Currently blocked IPs, by scenario
sum by (reason) (cs_active_decisions)

# Local and community decisions
sum by (origin, action) (cs_active_decisions)

# Attacks detected over 24 h, by scenario
sum by (name) (increase(cs_bucket_overflowed_total[24h]))

# Log lines not understood: is a parser missing?
sum by (source) (rate(cs_parser_hits_ko_total[5m]))

# Volume read per log source
sum by (source) (rate(cs_filesource_hits_total[5m]))

increase() does not count the very first overflow of a scenario: the series only exists from then on. For an exact count, also look at cs_alerts.

For Grafana, CrowdSec publishes ready-to-import dashboards in the crowdsecurity/grafana-dashboards repository (Overview, Insight, Details).

Alerting on health #

The worst incident is silent: CrowdSec reads nothing any more and nothing is blocked.

groups:
  - name: crowdsec
    rules:
      - alert: CrowdSecLogsNotRead
        expr: sum(rate(cs_filesource_hits_total[10m])) == 0
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "CrowdSec no longer reads any log line"
      - alert: CrowdSecLogsNotParsed
        expr: sum(rate(cs_parser_hits_ko_total[10m])) > 1
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "Many log lines are not understood"

4. Notifications and console #

Being notified live.

Notifications #

The slack, email, http and splunk plugins are configured in /etc/crowdsec/notifications/, for example http.yaml:

type: http
name: http_default
log_level: info
format: |
  {{ range . -}}
  {{ .Scenario }} : {{ range .Decisions }}{{ .Type }} {{ .Value }} for {{ .Duration }}{{ end }}
  {{ end -}}
url: https://example.com/alerts
method: POST

Then enable them in the profile, in profiles.yaml:

notifications:
  - http_default

crowdsecurity/ssh-bf : ban 203.0.113.50 for 4h

sudo cscli notifications list
sudo cscli notifications test http_default

Avoid noise. One notification per attack, on an exposed server, quickly adds up. Keep them for important scenarios, with a dedicated profile.

The console #

The CrowdSec console shows your instances, alerts and decisions online. It complements Prometheus, with nothing to host. You remain free not to use it.

Sources #

Commands, metrics and HTTP notification verified on CrowdSec 1.8.2.