Lab: Profile an application and link profiles to traces

What you will learn in this TP

• Send continuous profiles
• Read a flame graph
• Link spans and profiles

Introduction #

What you will learn in this section

• Send continuous profiles
• Read a flame graph
• Link spans and profiles

Context. A discount service is slow and CPU-hungry, but its traces only show a long span. You add the fourth signal to find the responsible line.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 45 min

A bottleneck found in the code through the span-profile link, then fixed and measured.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.

2. 2. A slow service

   Why? The code has a classic flaw: a regular expression recompiled and costly string building on every call.

   Install the Pyroscope SDK and its OpenTelemetry integration, then write remises.py: traces to Tempo, profiles to Pyroscope with the same service name, and the PyroscopeSpanProcessor.

3. 3. Read the flame graph

   Why? The flame graph shows where CPU is really spent, without adding a single span.

   In Explore > Pyroscope (or Drilldown > Profiles), select the remises application and the CPU type. Identify the widest plateaus and the share of code_valide and re.compile.

4. 4. From span to profile

   Why? The span processor tags samples with the span_id: you get the profile of one specific request.

   In Tempo, open a long calculer remise trace and use Profiles for this span. If the link does not appear, configure Trace to profiles in the Tempo data source (Pyroscope source, CPU type).

5. 5. Fix and compare

   Why? The profile proves an optimisation's effect instead of assuming it.

   Compile the expression once at module level and replace concatenation with ';'.join(). Restart, then compare both periods with Pyroscope's Diff view and span duration in Tempo.

6. 6. Place OpenTelemetry profiles

   Why? The Pyroscope SDK is a proven solution; the OTLP Profiles signal aims to unify collection but is not stable yet.

   Check the profiles signal status page on opentelemetry.io and the OpenTelemetry eBPF profiler documentation. Write in three lines what replacing the SDK with the eBPF profiler and a Collector would take, then stop the stack.

Difficulty level : ●●●○○ (3/5)