1. The fourth signal #
What metrics, logs and traces do not tell you.
From « what » to « which line of code » #
Available in: ✓ OSS ✓ Enterprise ✓ Cloud Pyroscope open source and Grafana Cloud Profiles
Metrics say a service uses 4 CPUs, traces say which call is slow, logs tell what happened. None of them says which function consumes that CPU or allocates that memory. That is the role of profiling.
Continuous profiling constantly samples the call stack of processes in production (for example 100 times per second) with low overhead (often 1 to 3% CPU). You can then go back to yesterday's spike, compare two versions or find the most expensive function across a fleet.
| Signal | Question | Grafana tool |
|---|---|---|
| Metrics | How much? Is it normal? | Prometheus, Mimir |
| Logs | What happened? | Loki |
| Traces | Where was time spent between services? | Tempo |
| Profiles | Which function consumes CPU or memory? | Pyroscope |
Good to know: Pyroscope was born in 2020 as an independent project, acquired by Grafana Labs in 2023 and merged with Grafana Phlare. It is open source (AGPLv3). OpenTelemetry is also standardizing profiles as a signal in their own right (OTLP profiles protocol, still in development).
2. Profile types #
CPU, memory, locks, goroutines.
The main types #
| Type | Measures | Languages |
|---|---|---|
CPU (process_cpu) | CPU time spent in each function | All |
Wall clock (wall) | Real time, including waits | Java, Python, .NET, Ruby |
Allocations (alloc_objects, alloc_space) | What is allocated (GC pressure) | Go, Java, .NET |
Memory in use (inuse_objects, inuse_space) | What stays in memory (leaks) | Go |
Locks (mutex, block, lock) | Time waiting on locks and channels | Go, Java, .NET |
| Goroutines | Number of goroutines per stack | Go |
In Pyroscope, a profile type is identified by a string such as process_cpu:cpu:nanoseconds:cpu:nanoseconds (name:sample type:unit:period type:unit).
3. Reading a flame graph #
The reference visualization.
Principle #
- Each rectangle is a function; its children (below it in Grafana) are the functions it calls.
- The width is the share of samples in which this function was on the stack: the wider, the more it costs.
- Horizontal order has no temporal meaning (sorted alphabetically or by size).
- Self time is the part of a function not covered by any child: it is the function's own code.
Good practice: Look for « plateaus »: a wide rectangle at the very bottom of a stack is a function that actually does work. Grafana's « Top table » sorts functions by self time: it is often the fastest starting point.
4. Pyroscope in the stack #
Collection, storage, exploration.
Three ways to send profiles #
| Mode | Principle | When to use it |
|---|---|---|
| SDK (push) | The Pyroscope library in the application sends its profiles | Business labels, span profiles, languages without pprof |
| Alloy (pull) | Alloy collects exposed pprof endpoints (Go, Rust...) | Go applications already instrumented with net/http/pprof |
| Alloy eBPF | Alloy profiles all processes on the host via the kernel | Profiling a whole fleet without changing code |
