Transforming with VRL and testing Vector pipelines

1. VRL basics #

A safe, fast language.

Syntax #

# Le point (.) est l'événement courant
.service = "boutique"
.status = to_int!(.status)                      # « ! » : erreur fatale si la conversion échoue

# Gestion explicite des erreurs
parsed, err = parse_json(.message)
if err == null {
  . = merge(., parsed)
  del(.message)
} else {
  .parse_error = err
}

# Conditions
if .status >= 500 { .severity = "error" } else if .status >= 400 { .severity = "warning" } else { .severity = "info" }

Good to know: VRL is checked at compile time: a function that can fail must be handled (! or assigning the error). A configuration that starts will not crash on an unexpected event.

2. Recipes #

Common transformations.

Parsing an nginx log #

parsed = parse_regex!(.message, r'^(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)')
. = merge(., parsed)
.status = to_int!(.status)
.bytes = to_int!(.bytes)
.timestamp = parse_timestamp!(.ts, "%d/%b/%Y:%H:%M:%S %z")
del(.ts)

Masking personal data #

.message = redact(.message, filters: [r'\b[\w.+-]+@[\w-]+\.[\w.]+\b', "us_social_security_number"])
.client_ip = ip_cidr_contains!("10.0.0.0/8", .ip) ? "interne" : "externe"
del(.authorization)

Good practice: Mask as close to the source as possible: an email that reaches Elasticsearch or Loki ends up in backups and indices for the whole retention period.

Routing and producing metrics #

transforms:
  aiguillage:
    type: route
    inputs: [parse]
    route:
      securite: .source == "auth"
      lent: .duration_ms > 1000
  requetes_metriques:
    type: log_to_metric
    inputs: [parse]
    metrics:
      - type: counter
        field: status
        name: http_requests_total
        tags: { status: "{{ status }}", method: "{{ method }}" }
sinks:
  prom:
    type: prometheus_exporter
    inputs: [requetes_metriques]
    address: 0.0.0.0:9598

3. Unit tests #

Configuration tested like code.

vector test #

tests:
  - name: analyse nginx
    inputs:
      - insert_at: parse
        type: log
        log_fields:
          message: '10.0.0.1 - - [19/Oct/2026:10:00:00 +0200] "GET /panier HTTP/1.1" 502 120'
    outputs:
      - extract_from: parse
        conditions:
          - type: vrl
            source: |
              assert_eq!(.status, 502)
              assert_eq!(.path, "/panier")

vector test vector.yaml

Good practice: Add vector validate and vector test to the configuration repository's CI: a broken regex no longer reaches production.