RousselTMLEARNING

Article · OpenTelemetry: From Discovery to Expertise

Profiles, the fourth signal

1. Why a fourth signal #

What traces do not tell you.

Continuous profiling #

A trace shows that a span lasted 2.3 seconds, but not which line of code consumed that time. Profiling regularly samples (for example 20 to 100 times per second) the call stacks of processes to measure where CPU, allocated memory, locks or I/O go. Continuous profiling does this permanently in production, with an overhead of around 1 to 2%.

Profile typeAnswers
CPUwhich functions consume the processor?
Memory allocationwho allocates the most (garbage collector pressure)?
Memory in use (heap)who holds on to memory (leak)?
Locks / mutexwhere do threads wait?
Wall clockwhere does real time go, waits included?

Reading a flame graph #

Flame graph
Here, compiling a regular expression on every call costs 26% of the CPU.
  • The horizontal axis is not time: it is the share of samples, sorted alphabetically.
  • Each level is a function call; children are above their parent (or below, depending on the tool).
  • A wide bar without children (« plateau ») is a function that consumes resources itself: it is the optimization target.

2. Profiles in OpenTelemetry #

A signal being standardized.

What exists #

Building blockStatus (to check on opentelemetry.io)
Data model and OTLPprofiles signal added to OTLP, inspired by pprof, with stack deduplication and a link to trace_id/span_id; development/alpha status
eBPF profilersystem profiler donated to the project by Elastic in 2024: profiles all processes on a host without instrumentation (Go, Java, Python, Node.js, .NET, Rust, C/C++…)
Collectorprofile receivers and exporters behind a feature gate; the eBPF profiler distributed as a Collector
Language SDKsno stable Profiles API yet: agents or the tools' SDKs are used (Pyroscope, async-profiler…)
BackendsGrafana Pyroscope, Elastic Universal Profiling, Datadog, Polar Signals… accept or announce OTLP Profiles
Common mistake: The profile signal is not yet stable: formats and options may change from one version to the next. In production, use your backend's proven integrations and follow the evolution of OTLP Profiles.

Enabling profiles in a recent Collector #

# feature gate à passer au démarrage du Collector (nom susceptible d'évoluer)
otelcol-contrib --config config.yaml --feature-gates=service.profilesSupport

# config.yaml : un pipeline de profils comme les autres
service:
  pipelines:
    profiles:
      receivers: [otlp]
      exporters: [otlp/pyroscope]

3. Linking profiles and traces #

The real gain: correlation.

Span profiles #

From span to profile
From Grafana Tempo, the « Profiles for this span » button opens the span's flame graph.

The profiler tags each sample with the active span_id. It then becomes possible to display the profile of one specific slow request rather than that of the whole process. With Pyroscope, a span processor adds this link:

import pyroscope
from pyroscope.otel import PyroscopeSpanProcessor

pyroscope.configure(application_name="checkout",            # identique à service.name
                    server_address="http://pyroscope:4040",
                    tags={"env": "production"})
provider.add_span_processor(PyroscopeSpanProcessor())       # étiquette les profils avec le span_id

Use cases #

  • Explain a latency regression between two versions (comparing flame graphs).
  • Reduce the cloud bill: 10% less CPU across 200 instances.
  • Find a memory leak before the forced restart (OOM).
  • Understand a slow span without adding manual spans everywhere.

4. Setting it up #

Starting simple.

Approach #

  1. Choose a profiles backend (Pyroscope is part of the Grafana stack).
  2. Profile without instrumentation using an eBPF agent (OpenTelemetry profiler, Alloy pyroscope.ebpf) to get a view of the whole fleet.
  3. Add profiling SDKs to critical services for memory profiles and span-profile correlation.
  4. Use the same service.name and resource attributes as the other signals.
  5. Follow the stabilization of OTLP Profiles to unify collection in the Collector.
Good to know: The Grafana Pyroscope course details installation, SDKs and profile analysis.

📄 Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces →

My progress

  • ⏱ Estimated time : ≈ 7 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

Continuous profiling with Grafana Pyroscope: why and how

The fourth signal of observability: what continuous profiling is, profile types (CPU, memory, loc...

Continuous profiling with eBPF: Parca, Pyroscope and the OpenTelemetry profiler

Continuously profile every process on a host with eBPF: principle, Parca and parca-agent, Pyrosco...

Linking traces and profiles: span profiles with Tempo and Pyroscope

Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

Analyzing profiles in Grafana: flame graphs, comparisons and diagnostics

A method for analyzing profiles in Grafana: Profiles Drilldown, flame graph and top table, compar...

Logs with OpenTelemetry: data model, appenders, correlation and collection

Master the log signal: the LogRecord data model, the Logs Bridge API and appenders (Logback, Log4...

Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack

A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...

OpenTelemetry traces in depth: spans, status, events, links and sampling

Master the trace signal: anatomy of a distributed trace, span and SpanKind, attributes, status an...

Correlating traces, logs, metrics and profiles in Grafana

Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!