Continuous profiling

1. The fourth signal #

What metrics, logs and traces do not tell you.

From « what » to « which line of code » #

Available in: ✓ OSS ✓ Enterprise ✓ Cloud Pyroscope open source and Grafana Cloud Profiles

Metrics say a service uses 4 CPUs, traces say which call is slow, logs tell what happened. None of them says which function consumes that CPU or allocates that memory. That is the role of profiling.

Continuous profiling constantly samples the call stack of processes in production (for example 100 times per second) with low overhead (often 1 to 3% CPU). You can then go back to yesterday's spike, compare two versions or find the most expensive function across a fleet.

Good to know: Pyroscope was born in 2020 as an independent project, acquired by Grafana Labs in 2023 and merged with Grafana Phlare. It is open source (AGPLv3). OpenTelemetry is also standardizing profiles as a signal in their own right (OTLP profiles protocol, still in development).

2. Profile types #

CPU, memory, locks, goroutines.

The main types #

In Pyroscope, a profile type is identified by a string such as process_cpu:cpu:nanoseconds:cpu:nanoseconds (name:sample type:unit:period type:unit).

3. Reading a flame graph #

The reference visualization.

Principle #

• Each rectangle is a function; its children (below it in Grafana) are the functions it calls.
• The width is the share of samples in which this function was on the stack: the wider, the more it costs.
• Horizontal order has no temporal meaning (sorted alphabetically or by size).
• Self time is the part of a function not covered by any child: it is the function's own code.

Good practice: Look for « plateaus »: a wide rectangle at the very bottom of a stack is a function that actually does work. Grafana's « Top table » sorts functions by self time: it is often the fastest starting point.

4. Pyroscope in the stack #

Collection, storage, exploration.

Three ways to send profiles #