1. Why a fourth signal #
What traces do not tell you.
Continuous profiling #
A trace shows that a span lasted 2.3 seconds, but not which line of code consumed that time. Profiling regularly samples (for example 20 to 100 times per second) the call stacks of processes to measure where CPU, allocated memory, locks or I/O go. Continuous profiling does this permanently in production, with an overhead of around 1 to 2%.
| Profile type | Answers |
|---|---|
| CPU | which functions consume the processor? |
| Memory allocation | who allocates the most (garbage collector pressure)? |
| Memory in use (heap) | who holds on to memory (leak)? |
| Locks / mutex | where do threads wait? |
| Wall clock | where does real time go, waits included? |
Reading a flame graph #
- The horizontal axis is not time: it is the share of samples, sorted alphabetically.
- Each level is a function call; children are above their parent (or below, depending on the tool).
- A wide bar without children (« plateau ») is a function that consumes resources itself: it is the optimization target.
2. Profiles in OpenTelemetry #
A signal being standardized.
What exists #
| Building block | Status (to check on opentelemetry.io) |
|---|---|
| Data model and OTLP | profiles signal added to OTLP, inspired by pprof, with stack deduplication and a link to trace_id/span_id; development/alpha status |
| eBPF profiler | system profiler donated to the project by Elastic in 2024: profiles all processes on a host without instrumentation (Go, Java, Python, Node.js, .NET, Rust, C/C++…) |
| Collector | profile receivers and exporters behind a feature gate; the eBPF profiler distributed as a Collector |
| Language SDKs | no stable Profiles API yet: agents or the tools' SDKs are used (Pyroscope, async-profiler…) |
| Backends | Grafana Pyroscope, Elastic Universal Profiling, Datadog, Polar Signals… accept or announce OTLP Profiles |
Common mistake: The profile signal is not yet stable: formats and options may change from one version to the next. In production, use your backend's proven integrations and follow the evolution of OTLP Profiles.
Enabling profiles in a recent Collector #
# feature gate à passer au démarrage du Collector (nom susceptible d'évoluer)
otelcol-contrib --config config.yaml --feature-gates=service.profilesSupport
# config.yaml : un pipeline de profils comme les autres
service:
pipelines:
profiles:
receivers: [otlp]
exporters: [otlp/pyroscope]3. Linking profiles and traces #
The real gain: correlation.
Span profiles #
The profiler tags each sample with the active span_id. It then becomes possible to display the profile of one specific slow request rather than that of the whole process. With Pyroscope, a span processor adds this link:
import pyroscope
from pyroscope.otel import PyroscopeSpanProcessor
pyroscope.configure(application_name="checkout", # identique à service.name
server_address="http://pyroscope:4040",
tags={"env": "production"})
provider.add_span_processor(PyroscopeSpanProcessor()) # étiquette les profils avec le span_idUse cases #
- Explain a latency regression between two versions (comparing flame graphs).
- Reduce the cloud bill: 10% less CPU across 200 instances.
- Find a memory leak before the forced restart (OOM).
- Understand a slow span without adding manual spans everywhere.
4. Setting it up #
Starting simple.
Approach #
- Choose a profiles backend (Pyroscope is part of the Grafana stack).
- Profile without instrumentation using an eBPF agent (OpenTelemetry profiler, Alloy
pyroscope.ebpf) to get a view of the whole fleet. - Add profiling SDKs to critical services for memory profiles and span-profile correlation.
- Use the same
service.nameand resource attributes as the other signals. - Follow the stabilization of OTLP Profiles to unify collection in the Collector.
Good to know: The Grafana Pyroscope course details installation, SDKs and profile analysis.
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Quels attributs utiliser pour nommer mes spans HTTP (semconv) ?✓ solved · 1 reply(ies)
- OpenTelemetry Collector : mes traces arrivent mais pas mes métriques✓ solved · 2 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
Continuous profiling with Grafana Pyroscope: why and how
The fourth signal of observability: what continuous profiling is, profile types (CPU, memory, loc...
Continuous profiling with eBPF: Parca, Pyroscope and the OpenTelemetry profiler
Continuously profile every process on a host with eBPF: principle, Parca and parca-agent, Pyrosco...
Linking traces and profiles: span profiles with Tempo and Pyroscope
Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...
Correlating traces, metrics, logs and profiles with OpenTelemetry
Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...
Analyzing profiles in Grafana: flame graphs, comparisons and diagnostics
A method for analyzing profiles in Grafana: Profiles Drilldown, flame graph and top table, compar...
Logs with OpenTelemetry: data model, appenders, correlation and collection
Master the log signal: the LogRecord data model, the Logs Bridge API and appenders (Logback, Log4...
Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack
A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...
OpenTelemetry traces in depth: spans, status, events, links and sampling
Master the trace signal: anatomy of a distributed trace, span and SpanKind, attributes, status an...
Correlating traces, logs, metrics and profiles in Grafana
Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...