Metrics and tags

1. Unified tagging #

Three tags to link everything.

env, service, version #

Datadog links the metrics, traces, logs and profiles of a service through three reserved tags: env, service and version. You set them once, at the source:

# Conteneur Docker
docker run -e DD_ENV=prod -e DD_SERVICE=checkout -e DD_VERSION=2.4.1 \
  -l com.datadoghq.tags.env=prod -l com.datadoghq.tags.service=checkout -l com.datadoghq.tags.version=2.4.1 ...

# Pod Kubernetes
metadata:
  labels:
    tags.datadoghq.com/env: prod
    tags.datadoghq.com/service: checkout
    tags.datadoghq.com/version: "2.4.1"

Good practice: With version, a service's page automatically compares deployments (latency, errors): deployment tracking with no effort.

2. Querying metrics #

Query syntax.

Queries #

avg:system.cpu.user{env:prod,service:checkout} by {host}
sum:trace.http.request.errors{service:checkout}.as_count() / sum:trace.http.request.hits{service:checkout}.as_count()
top(avg:kubernetes.cpu.usage.total{kube_namespace:boutique} by {pod_name}, 10, 'mean', 'desc')
anomalies(avg:aws.elb.latency{env:prod}, 'agile', 2)

3. Custom metrics and cost #

DogStatsD and cardinality.

DogStatsD #

# Envoyer une métrique en UDP (port 8125 de l'agent)
echo -n "boutique.commandes:1|c|#env:prod,service:checkout,moyen_paiement:carte" | nc -u -w1 localhost 8125
echo -n "boutique.panier.montant:129.9|d|#env:prod,service:checkout" | nc -u -w1 localhost 8125

from datadog import initialize, statsd
initialize(statsd_host="localhost", statsd_port=8125)
statsd.increment("boutique.commandes", tags=["moyen_paiement:carte"])
statsd.distribution("boutique.panier.montant", 129.9)

Common mistake: Each unique combination of name and tags is a billed custom metric. A user_id or order_id tag can create hundreds of thousands of series and multiply the bill.

Controlling cost #

• List the largest custom metrics (Metrics Summary, Usage pages).
• Use Metrics without Limits to keep only the tags actually queried.
• Prefer distributions to multiple gauges for percentiles.
• Do not replicate useless Prometheus metrics in Datadog (filter at collection).