1. OpenPipeline #
Processing before storing.
Journey #
OpenPipeline is the processing layer of the new platform: data (logs, events, spans, metrics depending on the version) is routed, transformed and enriched there before being stored in Grail. What is not stored is neither billed for retention nor exposed.
2. Processing #
Parse, enrich, mask.
Processors #
// Extraire les champs d'un log d'accès (processeur DQL)
parse content, "IPADDR:client.ip ' - - [' HTTPDATE:ts '] \"' LD:http.method ' ' LD:http.path ' ' LD '\" ' INT:http.status ' ' INT:bytes"
| fieldsAdd log.type = "nginx-access"
// Masquer une adresse e-mail
| fieldsAdd content = replacePattern(content, "LD '@' LD '.' ALPHA{2,}", "[email]")| Stage | Examples |
|---|---|
| Routing | Logs with k8s.namespace.name == "payment" to the « payment » pipeline |
| Processing | Parsing (DPL), adding fields, removing unneeded fields, normalizing severity |
| Masking | Credit cards, emails, tokens: replaced before storage |
| Extraction | Metric « number of 5xx errors per service »; Davis event on a critical pattern |
| Storage | logs_payment_365d bucket for auditing, default bucket for the rest |
Common mistake: Masking in OpenPipeline happens after sending: for data that must never leave the host, also configure masking at capture in OneAgent (log masking rules).
3. Buckets and retention #
Paying for the right duration.
Rules #
- A Grail bucket has a retention period; audit logs can be kept for a year, debug logs for a few days.
- Permissions can apply to buckets: security logs are only visible to the security team.
- Extracting a metric from a log and keeping the log only for a short time greatly reduces costs for long-term dashboards.
Good practice: Measure volume per source (fetch logs | summarize count(), by: {log.source}) before deciding on pipelines: a few sources often make up most of the volume.Good to know: Dynatrace evolves very quickly (new platform, Grail, apps): check screen names and API versions in your environment's documentation.
