Fluent Bit: pipeline inputs in detail

1. The role of inputs #

The entry point for data.

Principle #

An input collects data: it reads a file, listens on a port, queries the system or receives a protocol. Each record gets a tag (for example kube.var.log.containers.api-xyz.log): filters and outputs use it, through Match, to decide what they process.

2. Reading files with tail #

The most used input.

Configuration #

pipeline:
  inputs:
    - name: tail
      tag: app.*
      path: /var/log/app/*.log
      exclude_path: /var/log/app/*.gz
      db: /var/lib/fluent-bit/tail-app.db      # positions de lecture
      read_from_head: false                    # nouveau fichier : partir de la fin
      refresh_interval: 10                     # recherche de nouveaux fichiers (s)
      rotate_wait: 30                          # continuer à lire l'ancien fichier après rotation (s)
      skip_long_lines: on
      multiline.parser: docker, cri, java      # recoller les lignes d'une exception
      mem_buf_limit: 50MB
      storage.type: filesystem

Points of attention #

• path accepts wildcards; a tag with * is completed with the file path (dots instead of slashes).
• Multiline is configured right in the input: docker and cri rebuild lines split by the runtime; java, python, go group stack traces.
• The DB file must be on a persistent volume (hostPath on Kubernetes).

3. Other useful inputs #

Network, journald, OpenTelemetry.

Examples #

pipeline:
  inputs:
    - name: systemd
      tag: host.journal
      systemd_filter: _SYSTEMD_UNIT=nginx.service
      read_from_tail: on
    - name: syslog
      mode: udp
      listen: 0.0.0.0
      port: 5140
      parser: syslog-rfc5424
    - name: opentelemetry          # reçoit logs, métriques et traces OTLP/HTTP
      listen: 0.0.0.0
      port: 4318
    - name: forward                # depuis d'autres agents Fluent Bit ou Fluentd
      listen: 0.0.0.0
      port: 24224
    - name: node_exporter_metrics
      tag: host.metrics
      scrape_interval: 30

Memory and backpressure #

When an output slows down, data piles up. mem_buf_limit caps an input's memory: once reached, the input is paused (risk of loss for network inputs). With storage.type: filesystem, chunks overflow to disk and nothing is lost.

service:
  storage.path: /var/lib/fluent-bit/buffer
  storage.sync: normal
  storage.backlog.mem_limit: 50M

Good practice: For network inputs (syslog, http, forward), the disk buffer is essential: a client sending during a pause loses its messages.