Logs in Datadog

1. Collecting #

Agent and sources.

Configuration #

# datadog.yaml
logs_enabled: true

# conf.d/app.d/conf.yaml
logs:
  - type: file
    path: /var/log/app/*.log
    service: checkout
    source: python
    log_processing_rules:
      - type: mask_sequences
        name: masquer_cartes
        pattern: '\b\d{4}[ -]?\d{4}[ -]?\d{4}[ -]?\d{4}\b'
        replace_placeholder: "[CARTE MASQUÉE]"
      - type: exclude_at_match
        name: sans_healthcheck
        pattern: GET /health

Good practice: Masking and filtering in the agent avoids sending (and paying for) useless or sensitive data.

2. Structuring #

Pipelines and facets.

Pipelines #

Pipelines transform logs at ingestion: a grok parser extracts attributes, remappers designate the date, status and service, and other processors enrich (IP geolocation, user agent, lookup tables). Logs from known sources (nginx, postgres...) have ready-made integration pipelines.

regle_acces %{ip:network.client.ip} - - \[%{date("dd/MMM/yyyy:HH:mm:ss Z"):date}\] "%{word:http.method} %{notSpace:http.url} HTTP/%{number:http.version}" %{integer:http.status_code} %{integer:network.bytes_written}

An attribute becomes a facet (or a measure, for numbers) so it can be filtered and aggregated in the explorer:

service:checkout status:error @http.status_code:>=500 -@http.url:"/health"
@network.client.ip:10.0.* @duration:>2000000000

3. Controlling volume #

Logging without Limits.

Ingestion, indexes, archives #

• Exclusion filters per index (for example: index only 10% of status:info logs).
• Several indexes with different retentions (errors 30 days, debug 3 days).
• Daily quota per index to avoid surprises.

Common mistake: A log excluded from the index is not lost if it is archived, but it can no longer be viewed without rehydration. Decide with the teams what must remain immediately searchable.