Advanced logs: indexes, Flex and archives

1. Ingesting is not indexing #

The main cost lever.

Lifecycle #

2. Indexes and exclusion #

Several indexes, several retentions.

Configuration #

• An audit index at 30 days for security, an app index at 7 days, a debug index at 3 days.
• Indexes are evaluated in order: a log goes into the first index whose filter matches.
• An index's exclusion filters remove all or part (sampling) of the matching logs, for example 90% of health check status:info logs.
• Excluded logs remain visible in Live Tail, usable by Logs to Metrics and sent to archives.

Good practice: Before excluding, create a metric from the logs (Logs to Metrics): you keep the error count for 15 months without indexing the logs themselves.

3. Sensitive data #

Compliance.

Sensitive Data Scanner #

The Sensitive Data Scanner detects patterns (credit cards, emails, API keys, IBANs, social security numbers) in logs, traces and events, and masks, hashes or replaces them at ingestion, before indexing.

Common mistake: Masking on the Datadog side happens after sending: for data that must never leave your network, mask in the agent (log_processing_rules of type mask_sequences).

logs:
  - type: file
    path: /var/log/app/app.log
    service: shop
    source: python
    log_processing_rules:
      - type: mask_sequences
        name: masquer_emails
        replace_placeholder: "[email]"
        pattern: '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'