Policies, integrations, data streams and pipelines

1. Designing policies #

One policy per role.

Breakdown #

A policy groups what a set of hosts must collect. Split by role (web servers, databases, Windows desktops) and by environment through the namespace, rather than one giant policy for the whole fleet.

Policy settings #

• Default namespace, data output and monitoring output.
• Collection of the agent's own logs and metrics (monitoring).
• Unenrollment timeout for inactive agents, uninstall protection (Elastic Defend).

2. Integrations #

System, Nginx, Custom Logs.

What an integration contains #

An integration provides the collection configuration, the ingest pipelines that structure data (ECS), the mappings and dashboards. Its assets are installed in Elasticsearch and Kibana as soon as it is added to a policy.

Custom logs #

# Intégration « Custom Logs (Filestream) » : paramètres principaux
paths:
  - /var/log/monapp/*.log
data_stream.dataset: monapp
parsers:
  - multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'
      negate: true
      match: after
processors:
  - add_fields:
      target: ''
      fields:
        service.name: monapp

3. Customising ingestion #

Without changing managed assets.

The @custom pipeline #

Each integration data stream calls a <type>-<dataset>@custom pipeline if it exists. This is where to add your processing: it survives integration upgrades.

PUT _ingest/pipeline/logs-nginx.access@custom
{
  "processors": [
    { "set": { "field": "labels.equipe", "value": "web" } },
    { "drop": { "if": "ctx.url?.original?.startsWith('/health') == true" } }
  ]
}

Common mistake: Editing the integration's managed pipeline (logs-nginx.access-1.x.y) directly: the next upgrade overwrites it.

Variables and secrets #

Good practice: For integration passwords (databases, APIs), use Fleet secret fields: they are stored encrypted and do not appear in the exported policy.