Grafana Alloy: Continuous Profiling with Pyroscope

1. Introduction to continuous profiling #

Continuous profiling with Pyroscope.

The fourth signal #

Metrics say that the service consumes too much, traces say where the request loses time, and profiles say which function is responsible. A profile samples what the program does: call stack on the CPU, memory allocations, goroutines.

Why “continuous”? #

Classic profiling is started by hand once the problem is spotted, and it is often too late to reproduce it. Continuous profiling records all the time, with low overhead. You can then compare yesterday's version with today's, or examine an incident from last night.

Profile types #

Reading a flame graph #

Pyroscope displays profiles as a flame graph: each rectangle is a function, its width shows its share of the total. Look for wide rectangles: those are the ones to optimize. The horizontal axis is not time.

Pyroscope is the successor to Grafana Phlare. See the dedicated Grafana Pyroscope course to go further.

2. Setting up data collection #

Configuration of scraping and forwarding for Pyroscope.

Three ways to collect #

Collecting over HTTP #

In Go, exposing the pprof endpoints is enough:

import _ "net/http/pprof"

go func() {
    http.ListenAndServe(":6060", nil)
}()

Alloy queries them like metrics:

pyroscope.scrape "app" {
  targets         = [{ "__address__" = "app:6060", "service_name" = "boutique" }]
  scrape_interval = "15s"
  forward_to      = [pyroscope.write.backend.receiver]

  profiling_config {
    profile.process_cpu {
      enabled = true
    }
    profile.memory {
      enabled = true
    }
    profile.goroutine {
      enabled = true
    }
  }
}

pyroscope.write "backend" {
  endpoint {
    url = "http://pyroscope:4040"
  }
  external_labels = { env = "prod" }
}

Tested by profiling Alloy itself, which exposes pprof on its HTTP port: in just over 30 seconds, the backend received 6 pushes on /push.v1.PusherService/Push.

The service name matters: add service_name to your targets: it is how you find the application in Pyroscope. And never expose the pprof port on the Internet.

Without touching the code: eBPF #

discovery.process "tous" {
}

pyroscope.ebpf "hote" {
  targets    = discovery.process.tous.targets
  forward_to = [pyroscope.write.backend.receiver]
}

pyroscope.write "backend" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

pyroscope.ebpf profiles processes from the kernel, in any language. It requires running Alloy as root (or in a privileged container), and it only measures CPU. We validated this configuration with alloy validate without running it: the test requires root privileges.

Choosing #

You can link profiles to traces: add the same labels (service_name, env) on both sides to navigate from one to the other in Grafana.