Advanced APM: sampling and profiler

1. Ingestion and retention #

Two levels of control.

Principle #

• Trace metrics (hits, errors, latency per resource) are computed on all traffic, whatever the sampling.
• Sampling rules (DD_TRACE_SAMPLING_RULES variable or remote configuration) decide which traces are ingested.
• Retention filters choose the spans indexed for 15 days: by default errors, high latencies and a diverse sample.

export DD_TRACE_SAMPLING_RULES='[{"service":"shop","name":"http.request","sample_rate":0.2},{"service":"shop","resource":"GET /health","sample_rate":0}]' 

2. Profiler and dynamic instrumentation #

Beyond traces.

Tools #

3. Software Catalog #

Who owns what.

Service ownership #

The Software Catalog (formerly Service Catalog) lists services with their team, on-call, repositories, runbooks and criticality level, described in a file versioned with the code:

# service.datadog.yaml
apiVersion: v3
kind: service
metadata:
  name: shop
  owner: team-commerce
  links:
    - { name: Runbook, type: runbook, url: https://wiki.example.com/shop }
spec:
  lifecycle: production
  tier: "1"

Good practice: A service without an owner is an incident without anyone responsible: make the catalog entry a prerequisite for going to production.