CrowdSec observability

1. Answering the right questions #

The tools available.

Overview #

2. With cscli #

The immediate view.

Tracking banned IPs #

sudo cscli decisions list
sudo cscli decisions list --origin crowdsec
sudo cscli decisions list --origin CAPI
sudo cscli decisions list -o json

| ID | Source   | Scope:Value     | Reason               | Action | Events | expiration |
| 1  | crowdsec | Ip:203.0.113.50 | crowdsecurity/ssh-bf | ban    | 6      | 3h59m46s   |

The Source column shows the origin: crowdsec (local detection), cscli (manual), CAPI (community).

Analyzing an attack #

sudo cscli alerts list --since 24h
sudo cscli alerts inspect 3 --details

- ID           : 3
 - Machine      : test
 - Simulation   : false
 - Remediation  : true
 - Reason       : crowdsecurity/ssh-bf
 - Events Count : 6
 - Scope:Value  : Ip:203.0.113.50

Measuring actual blocking #

A decision does not say whether anything was blocked. Only the bouncer knows: cscli metrics show bouncers shows, per origin, the number of active addresses and the blocked volume (CrowdSec 1.6.3 and firewall bouncer 0.0.30 minimum).

sudo cscli metrics show bouncers
sudo nft list table ip crowdsec   # firewall counters

Troubleshooting #

3. Prometheus and Grafana #

History and dashboards.

Exposing metrics #

In config.yaml:

prometheus:
  enabled: true
  level: full            # none, aggregated or full
  listen_addr: 0.0.0.0   # 127.0.0.1 by default: open it for a remote Prometheus
  listen_port: 6060

curl -s http://127.0.0.1:6060/metrics | grep -E '^cs_(active_decisions|alerts)'
# cs_active_decisions{action="ban",origin="crowdsec",reason="crowdsecurity/ssh-bf"} 1
# cs_alerts{kind="crowdsec",reason="crowdsecurity/ssh-bf"} 1

scrape_configs:
  - job_name: crowdsec
    static_configs:
      - targets: ["crowdsec:6060"]

Detail level. The aggregated level notably removes cs_active_decisions, cs_alerts and per-bouncer metrics. Keep full to track blocked IPs.

Useful metrics #

PromQL queries #

# Currently blocked IPs, by scenario
sum by (reason) (cs_active_decisions)

# Local and community decisions
sum by (origin, action) (cs_active_decisions)

# Attacks detected over 24 h, by scenario
sum by (name) (increase(cs_bucket_overflowed_total[24h]))

# Log lines not understood: is a parser missing?
sum by (source) (rate(cs_parser_hits_ko_total[5m]))

# Volume read per log source
sum by (source) (rate(cs_filesource_hits_total[5m]))

increase() does not count the very first overflow of a scenario: the series only exists from then on. For an exact count, also look at cs_alerts.

For Grafana, CrowdSec publishes ready-to-import dashboards in the crowdsecurity/grafana-dashboards repository (Overview, Insight, Details).

Alerting on health #

The worst incident is silent: CrowdSec reads nothing any more and nothing is blocked.

groups:
  - name: crowdsec
    rules:
      - alert: CrowdSecLogsNotRead
        expr: sum(rate(cs_filesource_hits_total[10m])) == 0
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "CrowdSec no longer reads any log line"
      - alert: CrowdSecLogsNotParsed
        expr: sum(rate(cs_parser_hits_ko_total[10m])) > 1
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "Many log lines are not understood"

4. Notifications and console #

Being notified live.

Notifications #

The slack, email, http and splunk plugins are configured in /etc/crowdsec/notifications/, for example http.yaml:

type: http
name: http_default
log_level: info
format: |
  {{ range . -}}
  {{ .Scenario }} : {{ range .Decisions }}{{ .Type }} {{ .Value }} for {{ .Duration }}{{ end }}
  {{ end -}}
url: https://example.com/alerts
method: POST

Then enable them in the profile, in profiles.yaml:

notifications:
  - http_default

crowdsecurity/ssh-bf : ban 203.0.113.50 for 4h

sudo cscli notifications list
sudo cscli notifications test http_default

Avoid noise. One notification per attack, on an exposed server, quickly adds up. Keep them for important scenarios, with a dedicated profile.

The console #

The CrowdSec console shows your instances, alerts and decisions online. It complements Prometheus, with nothing to host. You remain free not to use it.

Sources #

• CrowdSec : Prometheus
• CrowdSec : notifications
• crowdsecurity/grafana-dashboards

Commands, metrics and HTTP notification verified on CrowdSec 1.8.2.