RousselTMLEARNING

Observability

Datadog best practices: tags, costs, alerts and governance

Checklist for using Datadog effectively: unified tagging, cost control (hosts, logs, custom metrics, traces), useful alerts, configuration as code and OpenTelemetry.

1. Foundation #

Getting started right.

Principles #

  • Site chosen according to data location (EU for GDPR).
  • Unified tagging env, service, version everywhere, plus team.
  • Agent deployed by the Kubernetes operator or configuration management, with a controlled version.
  • API keys per use case, named and revocable application keys.

2. Costs #

The bill can be managed.

Levers #

ItemLever
Hosts and containersDo not install the full agent where it is not needed
LogsFilter in the agent, exclusion filters, short-retention indexes, archives
Custom metricsNo high-cardinality tags, Metrics without Limits
APMSampling and retention filters
TrackingUsage page, alerts on usage metrics (datadog.estimated_usage.*)
Good practice: Create monitors on datadog.estimated_usage.logs.ingested_bytes and datadog.estimated_usage.metrics.custom: a drift is detected within hours, not on the bill.

3. Alerts and governance #

Sustainable and shared.

Rules #

  • Multi-alert monitors per service, with a runbook and a recipient team.
  • SLOs for critical journeys, burn rate for on-call.
  • Monitors, dashboards and SLOs in Terraform, reviewed in pull requests.
  • OpenTelemetry instrumentation when possible, to keep freedom of choice.

Reference diagram and quick check #

Checklist.

Checklist #

  1. Site and keys configured
  2. Unified tagging verified
  3. Log pipelines and indexes under control
  4. APM with logs-traces correlation
  5. Monitors and SLOs in Terraform
  6. Usage alerts

Related articles

Datadog platform architecture: sites, organizations, keys and data model

Understand Datadog's architecture: data sources (agent, cloud integrations, SDKs, OpenTelemetry), regional si…

Datadog on Kubernetes: Operator, Cluster Agent and Autodiscovery

Monitor Kubernetes with Datadog: the Datadog Operator and the DatadogAgent resource, Cluster Agent, annotatio…

Datadog dashboards and notebooks: widgets, template variables and best practices

Design useful Datadog dashboards: widget types, template variables, groups, Powerpacks, formulas, investigati…

Keep learning

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!