Vector: sources in detail

1. The role of sources #

The starting point of the topology.

Principle #

2. Examples #

The most common sources.

Configuration #

sources:
  fichiers_app:
    type: file
    include: ["/var/log/app/*.log"]
    exclude: ["/var/log/app/*.gz"]
    read_from: end                     # nouveau fichier : partir de la fin
    ignore_older_secs: 86400
    multiline:                          # recoller une trace de pile
      start_pattern: '^\d{4}-\d{2}-\d{2}'
      mode: halt_before
      condition_pattern: '^\d{4}-\d{2}-\d{2}'
      timeout_ms: 1000
  kube:
    type: kubernetes_logs              # lit /var/log/pods et ajoute pod, namespace, labels
    extra_label_selector: "logging!=off"
  journal:
    type: journald
    include_units: [nginx, sshd]
  syslog_reseau:
    type: syslog
    mode: udp
    address: 0.0.0.0:5140
  otlp:
    type: opentelemetry
    grpc: { address: 0.0.0.0:4317 }
    http: { address: 0.0.0.0:4318 }
  hote:
    type: host_metrics
    collectors: [cpu, memory, disk, filesystem, network]
  vector_interne:
    type: internal_metrics

3. Positions and acknowledgements #

Losing nothing.

Reliability #

• File sources remember their position in data_dir (to place on a persistent volume) and identify a file by a fingerprint of its first bytes, which handles rotation.
• With acknowledgements: true on sinks, a source that supports it (HTTP, Kafka, files) only acknowledges or advances once the event has been delivered.
• The internal_metrics source exposes the throughput and errors of each component.

data_dir: /var/lib/vector
sinks:
  loki:
    type: loki
    inputs: [kube]
    endpoint: http://loki:3100
    acknowledgements:
      enabled: true

Good practice: Check that a source actually produces events with vector tap --outputs-of kube before writing the transformations.