What you will learn in this TP
- Send continuous profiles
- Read a flame graph
- Link spans and profiles
Introduction #
What you will learn in this section
- Send continuous profiles
- Read a flame graph
- Link spans and profiles
Context. A discount service is slow and CPU-hungry, but its traces only show a long span. You add the fourth signal to find the responsible line.
Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container) · Estimated time: 45 min
Hands-on #
Guided steps, with their solution.
1. Start the observability stack
Why? A single image provides an OTLP Collector and every backend: focus on the signal.
Startgrafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.2. A slow service
Why? The code has a classic flaw: a regular expression recompiled and costly string building on every call.
Install the Pyroscope SDK and its OpenTelemetry integration, then writeremises.py: traces to Tempo, profiles to Pyroscope with the same service name, and thePyroscopeSpanProcessor.3. Read the flame graph
Why? The flame graph shows where CPU is really spent, without adding a single span.
In Explore > Pyroscope (or Drilldown > Profiles), select theremisesapplication and the CPU type. Identify the widest plateaus and the share ofcode_valideandre.compile.4. From span to profile
Why? The span processor tags samples with the span_id: you get the profile of one specific request.
In Tempo, open a longcalculer remisetrace and use Profiles for this span. If the link does not appear, configure Trace to profiles in the Tempo data source (Pyroscope source, CPU type).5. Fix and compare
Why? The profile proves an optimisation's effect instead of assuming it.
Compile the expression once at module level and replace concatenation with';'.join(). Restart, then compare both periods with Pyroscope's Diff view and span duration in Tempo.6. Place OpenTelemetry profiles
Why? The Pyroscope SDK is a proven solution; the OTLP Profiles signal aims to unify collection but is not stable yet.
Check the profiles signal status page on opentelemetry.io and the OpenTelemetry eBPF profiler documentation. Write in three lines what replacing the SDK with the eBPF profiler and a Collector would take, then stop the stack.
Difficulty level : ●●●○○ (3/5)
My progress
Sign in to track your progress and take notes. Sign in
💬 Recent discussions about this course
- Quels attributs utiliser pour nommer mes spans HTTP (semconv) ?✓ solved · 1 reply(ies)
- OpenTelemetry Collector : mes traces arrivent mais pas mes métriques✓ solved · 2 reply(ies)
Ask a question about this lesson →See all discussions about this course
Recommended Articles
Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces
Understand OpenTelemetry's new profile signal: continuous profiling, flame graphs, data model and...
Linking traces and profiles: span profiles with Tempo and Pyroscope
Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...
Correlating traces, metrics, logs and profiles with OpenTelemetry
Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...
Continuous profiling with Grafana Pyroscope: why and how
The fourth signal of observability: what continuous profiling is, profile types (CPU, memory, loc...
Analyzing profiles in Grafana: flame graphs, comparisons and diagnostics
A method for analyzing profiles in Grafana: Profiles Drilldown, flame graph and top table, compar...
Continuous profiling with eBPF: Parca, Pyroscope and the OpenTelemetry profiler
Continuously profile every process on a host with eBPF: principle, Parca and parca-agent, Pyrosco...
Correlating traces, logs, metrics and profiles in Grafana
Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...
Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack
A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...
Tempo: generating metrics from traces (service graph, span metrics, exemplars)
Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...