Lab: Profile an application and link profiles to traces with correction

What you will learn in this TP

• Send continuous profiles
• Read a flame graph
• Link spans and profiles

Introduction #

What you will learn in this section

• Send continuous profiles
• Read a flame graph
• Link spans and profiles

Context. A discount service is slow and CPU-hungry, but its traces only show a long span. You add the fourth signal to find the responsible line.

Prerequisites: Docker and Python 3.10 or later; the grafana/otel-lgtm stack (Collector, Tempo, Loki, Prometheus, Pyroscope and Grafana in one container)  ·  Estimated time: 45 min

A bottleneck found in the code through the span-profile link, then fixed and measured.

Hands-on #

Guided steps, with their solution.

1. 1. Start the observability stack

   Why? A single image provides an OTLP Collector and every backend: focus on the signal.

   Start grafana/otel-lgtm, create a Python environment and point exporters to OTLP/HTTP port 4318.
   See the correction

   docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 -p 4040:4040 grafana/otel-lgtm
   python3 -m venv .venv && . .venv/bin/activate
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   # Grafana : http://localhost:3000 (admin / admin)

2. 2. A slow service

   Why? The code has a classic flaw: a regular expression recompiled and costly string building on every call.

   Install the Pyroscope SDK and its OpenTelemetry integration, then write remises.py: traces to Tempo, profiles to Pyroscope with the same service name, and the PyroscopeSpanProcessor.
   See the correction

   pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http pyroscope-io pyroscope-otel
   cat > remises.py <<'EOF'
   import random, re, time
   import pyroscope
   from pyroscope.otel import PyroscopeSpanProcessor
   from opentelemetry import trace
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   
   pyroscope.configure(application_name="remises", server_address="http://localhost:4040", tags={"env": "tp"})
   provider = TracerProvider(resource=Resource.create({"service.name": "remises"}))
   provider.add_span_processor(PyroscopeSpanProcessor())
   provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(provider)
   tracer = trace.get_tracer("tp.remises")
   
   def code_valide(code):
       motif = re.compile(r"^(PROMO|VIP)-[0-9]{4}" + "(-X)?" * random.randint(20, 40) + "$")   # recompilé à chaque appel
       return bool(motif.match(code))
   
   def calculer_remise(panier):
       texte = ""
       for article in panier:
           texte += f"{article};"                       # concaténation quadratique
       return sum(1 for a in panier if code_valide(a)) * 5
   
   while True:
       with tracer.start_as_current_span("calculer remise") as span:
           panier = [f"PROMO-{random.randint(1000, 9999)}" for _ in range(random.randint(200, 800))]
           span.set_attribute("shop.cart.items", len(panier))
           calculer_remise(panier)
       time.sleep(0.05)
   EOF
   python remises.py &

3. 3. Read the flame graph

   Why? The flame graph shows where CPU is really spent, without adding a single span.

   In Explore > Pyroscope (or Drilldown > Profiles), select the remises application and the CPU type. Identify the widest plateaus and the share of code_valide and re.compile.
   See the correction

   # requête Pyroscope
   process_cpu:cpu:nanoseconds:cpu:nanoseconds{service_name="remises"}
   # attendu : calculer_remise > code_valide > re.compile / sre_compile très larges

4. 4. From span to profile

   Why? The span processor tags samples with the span_id: you get the profile of one specific request.

   In Tempo, open a long calculer remise trace and use Profiles for this span. If the link does not appear, configure Trace to profiles in the Tempo data source (Pyroscope source, CPU type).
   See the correction

   { resource.service.name = "remises" && duration > 300ms }
   # span sélectionné > Profiles for this span : flame graph limité à ce span
   # vérifier l'attribut pyroscope.profile.id ajouté au span par le PyroscopeSpanProcessor

5. 5. Fix and compare

   Why? The profile proves an optimisation's effect instead of assuming it.

   Compile the expression once at module level and replace concatenation with ';'.join(). Restart, then compare both periods with Pyroscope's Diff view and span duration in Tempo.
   See the correction

   kill %1
   python - <<'EOF'
   s = open("remises.py").read()
   s = s.replace('def code_valide(code):\n    motif = re.compile(r"^(PROMO|VIP)-[0-9]{4}" + "(-X)?" * random.randint(20, 40) + "$")   # recompilé à chaque appel\n    return bool(motif.match(code))',
                 'MOTIF = re.compile(r"^(PROMO|VIP)-[0-9]{4}(-X)*$")\n\ndef code_valide(code):\n    return bool(MOTIF.match(code))')
   s = s.replace('    texte = ""\n    for article in panier:\n        texte += f"{article};"                       # concaténation quadratique\n', '    texte = ";".join(panier)\n')
   open("remises.py", "w").write(s)
   EOF
   python remises.py &
   # Tempo : avg_over_time des durées { name = "calculer remise" } ; Pyroscope : Diff avant / après

6. 6. Place OpenTelemetry profiles

   Why? The Pyroscope SDK is a proven solution; the OTLP Profiles signal aims to unify collection but is not stable yet.

   Check the profiles signal status page on opentelemetry.io and the OpenTelemetry eBPF profiler documentation. Write in three lines what replacing the SDK with the eBPF profiler and a Collector would take, then stop the stack.
   See the correction

   kill %1 ; docker rm -f lgtm
   # Pistes : profileur eBPF en DaemonSet (privilégié, noyau récent), Collector avec feature gate profiles,
   # exportateur OTLP vers Pyroscope, mêmes attributs de ressource que les traces

Difficulty level : ●●●○○ (3/5)