RousselTMLEARNING

TP · OpenTelemetry: From Discovery to Expertise

Lab: Profile an application and link profiles to traces

What you will learn in this TP
  • Send continuous profiles
  • Read a flame graph
  • Link spans and profiles

Introduction #

What you will learn in this section
  • Send continuous profiles
  • Read a flame graph
  • Link spans and profiles

Context. A discount service is slow and CPU-hungry, but its traces only show a long span. You add the fourth signal to find the responsible line.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 45 min

A bottleneck found in the code through the span-profile link, then fixed and measured.

Hands-on #

Guided steps, with their solution.

  1. 1. Start the observability stack

    Why? A single image provides an OTLP Collector and every backend: focus on the signal.

    Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
  2. 2. A slow service

    Why? The code has a classic flaw: a regular expression recompiled and costly string building on every call.

    Install the Pyroscope SDK and its OpenTelemetry integration, then write remises.py: traces to Tempo, profiles to Pyroscope with the same service name, and the PyroscopeSpanProcessor.
  3. 3. Read the flame graph

    Why? The flame graph shows where CPU is really spent, without adding a single span.

    In Explore > Pyroscope (or Drilldown > Profiles), select the remises application and the CPU type. Identify the widest plateaus and the share of code_valide and re.compile.
  4. 4. From span to profile

    Why? The span processor tags samples with the span_id: you get the profile of one specific request.

    In Tempo, open a long calculer remise trace and use Profiles for this span. If the link does not appear, configure Trace to profiles in the Tempo data source (Pyroscope source, CPU type).
  5. 5. Fix and compare

    Why? The profile proves an optimisation's effect instead of assuming it.

    Compile the expression once at module level and replace concatenation with ';'.join(). Restart, then compare both periods with Pyroscope's Diff view and span duration in Tempo.
  6. 6. Place OpenTelemetry profiles

    Why? The Pyroscope SDK is a proven solution; the OTLP Profiles signal aims to unify collection but is not stable yet.

    Check the profiles signal status page on opentelemetry.io and the OpenTelemetry eBPF profiler documentation. Write in three lines what replacing the SDK with the eBPF profiler and a Collector would take, then stop the stack.

Difficulty level : ●●●○○ (3/5)

My progress

  • ⏱ Estimated time : ≈ 45 min

Sign in to track your progress and take notes. Sign in

💬 Recent discussions about this course

Ask a question about this lesson →See all discussions about this course

Recommended Articles

Profiles, OpenTelemetry's fourth signal: continuous profiling and correlation with traces

Understand OpenTelemetry's new profile signal: continuous profiling, flame graphs, data model and...

Linking traces and profiles: span profiles with Tempo and Pyroscope

Go from a slow trace to the exact CPU profile of that span: span profiles in Go and Java, configu...

Correlating traces, metrics, logs and profiles with OpenTelemetry

Link the four signals to diagnose quickly: shared resource, trace_id in logs, exemplars on metric...

Continuous profiling with Grafana Pyroscope: why and how

The fourth signal of observability: what continuous profiling is, profile types (CPU, memory, loc...

Analyzing profiles in Grafana: flame graphs, comparisons and diagnostics

A method for analyzing profiles in Grafana: Profiles Drilldown, flame graph and top table, compar...

Continuous profiling with eBPF: Parca, Pyroscope and the OpenTelemetry profiler

Continuously profile every process on a host with eBPF: principle, Parca and parca-agent, Pyrosco...

Correlating traces, logs, metrics and profiles in Grafana

Link Tempo to logs (Loki), metrics (Prometheus) and profiles (Pyroscope): tracesToLogs, tracesToM...

Capstone project: the four signals of the OpenTelemetry Demo in the Grafana stack

A capstone project spanning the observability courses: deploy the OpenTelemetry Demo, send metric...

Tempo: generating metrics from traces (service graph, span metrics, exemplars)

Enable Tempo's metrics-generator: service graphs, RED span metrics, remote write to Prometheus or...

Support us

♥ SUPPORT US

Support RousselTM Learning

Courses, articles, labs and certificates stay free with no intrusive ads. A voluntary contribution, of any amount you choose, pays for hosting, lab environments and new content.

Contribute with PayPal Any amount · card or PayPal accountLearn more
QR code PayPal
Or scan this QR code with your phone

Thank you, every contribution counts!