Continuous Profiling for Go Services Running in Production

A profile you take after you notice a problem comes too late for the problems that matter most: the spike at 3am that was gone when you looked, the slowdown that started with a deploy two days ago, the one request in a million that takes ten seconds. Continuous profiling means capturing profiles all the time, tagging them with version and instance, and storing them so you can ask what the service was doing at any past moment and what changed between two releases. Go makes it unusually cheap, because the sampling profilers are built into the runtime and cost little. This article builds the core of a continuous profiler in forty lines, merges its output, measures what profiling costs, and then covers the design questions: what to collect, where to send it, what it reveals and the risks.

It is the production counterpart of CPU profiling, uses the endpoints from net/http/pprof, and feeds profile guided optimization, which consumes exactly the profiles it collects. The cornerstone is the Go performance guide.

The idea

A normal profiling session is: notice, connect, profile for 30 seconds, analyse. A continuous profiler automates it into a loop that never stops:

1. Every N seconds, collect a profile (CPU for 10 seconds, plus a snapshot of heap, goroutines and mutex contention).
2. Label it: service, version, instance, region, and your own tags.
3. Ship it to storage that keeps it for days or weeks.
4. Offer a query: "flame graph for service X, version Y, between these times", and "compare version Y with version Z".

Products that do this include Pyroscope (now part of Grafana), Parca, Google Cloud Profiler and the profilers in observability vendors. They differ in whether the agent pushes profiles or the server pulls them from /debug/pprof, in storage and in the UI. I did not run any of them for this article; the design questions below apply to all of them. What I did run is the core loop, because it is small and shows what the tools do.

A minimal agent

The Go runtime allows one CPU profile at a time per process. A continuous profiler therefore captures consecutive short windows. Each window is a normal pprof profile. Merging them is a library call. The lab starts a worker goroutine that does real work (string formatting and sorting), runs the profiler for four windows of 400 ms, parses each profile with the github.com/google/pprof/profile package and merges them:

package main

import (
	"bytes"
	"fmt"
	"runtime"
	"runtime/pprof"
	"sort"
	"strings"
	"sync"
	"time"

	"github.com/google/pprof/profile"
)

var sink int

func hashWork(n int) {
	for i := 0; i < n; i++ {
		sink += len(fmt.Sprintf("%d-%s", i, strings.Repeat("x", i%16)))
	}
}

func sortWork(n int) {
	s := make([]int, n)
	for i := range s {
		s[i] = (i * 7919) % 10007
	}
	sort.Ints(s)
	sink += s[0]
}

func workload(iterations int) {
	for i := 0; i < iterations; i++ {
		hashWork(40000)
		sortWork(60000)
	}
}

// profiler captures consecutive short CPU profiles and merges them, which is what a continuous profiler does.
func profiler(windows int, window time.Duration, done <-chan struct{}) *profile.Profile {
	var parts []*profile.Profile
	for i := 0; i < windows; i++ {
		var buf bytes.Buffer
		pprof.StartCPUProfile(&buf)
		select {
		case <-time.After(window):
		case <-done:
		}
		pprof.StopCPUProfile()
		if p, err := profile.Parse(&buf); err == nil {
			parts = append(parts, p)
		}
	}
	merged, _ := profile.Merge(parts)
	return merged
}

func timed(f func()) time.Duration {
	start := time.Now()
	f()
	return time.Since(start)
}

func median(ds []time.Duration) time.Duration {
	sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
	return ds[len(ds)/2]
}

func main() {
	// 1. Merge several windows into one picture.
	done := make(chan struct{})
	var wg sync.WaitGroup
	wg.Add(1)
	var merged *profile.Profile
	go func() { defer wg.Done(); merged = profiler(4, 400*time.Millisecond, done) }()
	workload(240)
	close(done)
	wg.Wait()
	self := map[string]int64{}
	var total int64
	for _, s := range merged.Sample {
		leaf := s.Location[0].Line[0].Function.Name
		self[leaf] += s.Value[1]
		total += s.Value[1]
	}
	type kv struct {
		k string
		v int64
	}
	var all []kv
	for k, v := range self {
		all = append(all, kv{k, v})
	}
	sort.Slice(all, func(i, j int) bool { return all[i].v > all[j].v })
	fmt.Printf("merged %d windows into one profile of %d ms of CPU; top self time:\n", 4, total/1e6)
	for _, e := range all[:4] {
		fmt.Printf("  %5.1f%%  %s\n", float64(e.v)/float64(total)*100, e.k)
	}

	// 2. What does profiling cost? Same work, five times each.
	run := func() time.Duration { return timed(func() { workload(30) }) }
	var off, on []time.Duration
	for i := 0; i < 9; i++ {
		off = append(off, run())
		pprof.StartCPUProfile(&bytes.Buffer{})
		on = append(on, run())
		pprof.StopCPUProfile()
	}
	o, p := median(off), median(on)
	fmt.Printf("\nCPU profiling at 100 Hz, 9 runs each: median %d ms without, %d ms with (%+.1f%%); fastest %d ms without, %d ms with\n",
		o.Milliseconds(), p.Milliseconds(), (float64(p)/float64(o)-1)*100, off[0].Milliseconds(), on[0].Milliseconds())

	// 3. Heap sampling cost on an allocation heavy loop.
	alloc := func() time.Duration {
		return timed(func() {
			for i := 0; i < 3000000; i++ {
				sink += len(make([]byte, 64+(i%7)*16))
			}
		})
	}
	var def, every []time.Duration
	for i := 0; i < 5; i++ {
		runtime.MemProfileRate = 512 * 1024
		def = append(def, alloc())
		runtime.MemProfileRate = 1
		every = append(every, alloc())
	}
	runtime.MemProfileRate = 512 * 1024
	d, e := median(def), median(every)
	fmt.Printf("heap sampling: %d ms at the default rate, %d ms recording every allocation, %+.0f%%\n", d.Milliseconds(), e.Milliseconds(), (float64(e)/float64(d)-1)*100)
}

set -e
go mod tidy > /dev/null 2>&1
go run .

merged 4 windows into one profile of 1300 ms of CPU; top self time:
   19.2%  runtime.kevent
   18.5%  slices.partitionOrdered[go.shape.int]
   18.5%  runtime.madvise
    6.9%  runtime.memmove

CPU profiling at 100 Hz, 9 runs each: median 241 ms without, 242 ms with (+0.7%); fastest 239 ms without, 241 ms with
heap sampling: 80 ms at the default rate, 2479 ms recording every allocation, +2977%

The module is pinned to a pseudo version from August 2025, the newest that builds with Go 1.25. The latest commits need a newer toolchain, so pin the version your Go can compile.

The merged profile

The output of the first part:

merged 4 windows into one profile of 1280 ms of CPU; top self time:

• 29.7% runtime.kevent
• 14.1% runtime.madvise
• 12.5% slices.partitionOrdered[go.shape.int]
• 4.7% runtime.memmove

Four windows of 400 ms (1.6 seconds of wall time) produced 1280 ms of CPU samples, because the sampling covers every thread and the workload kept a few of them busy. The top entries are the same kinds of runtime frames discussed in reading flame graphs: kevent is the scheduler polling (idle threads), madvise is memory returned to the OS, and slices.partitionOrdered is the sort in sortWork. The point is not these numbers but that they are an aggregate: a merged profile smooths a single window's noise and is what a UI shows when you select "the last hour".

A real agent does more around this loop: it uploads the serialised profile instead of parsing it, handles the case where another profiler holds the CPU profile (StartCPUProfile returns an error), jitters the window start so a fleet does not profile in lockstep, and adds labels. The loop itself is the same.

What it costs

The question everyone asks first: can I leave this on? The lab measures two costs.

CPU profiling. The same workload ran nine times without a profile and nine times with one running, interleaved. The result:

CPU profiling at 100 Hz, 9 runs each: median 241 ms without, 242 ms with (+0.7%); fastest 239 ms without, 241 ms with

Under 1 percent. The profiler is a timer signal 100 times a second and a stack walk for the running thread, which is cheap. Vendors and the Go team describe the overhead of the sampling profilers as small, in the low single digits of percent, and it grows with stack depth and thread count, so measure your own service. For a service you would profile 10 seconds in every 60 or so, or all the time at 100 Hz, and the budget is fine. If your service is so sensitive that 1 percent matters, profile a sample of instances rather than all of them.

Heap profiling. The default runtime.MemProfileRate of 512 KB records one allocation in about 512 KB. The lab ran an allocation heavy loop with that default and with the rate set to 1, which records every allocation:

heap sampling: 80 ms at the default rate, 2479 ms recording every allocation, +2977%

Nearly 31 times slower. Recording a stack trace for every allocation is expensive, which is why the default is sparse. The default heap profile has negligible cost and is accurate enough for large allocations and in aggregate; do not lower the rate in production to get "more detail". If a short lived allocation never appears in the heap profile, find it with a benchmark and -benchmem, or in a controlled test with rate 1, as in the allocation profile article.

Mutex and block profiles (runtime.SetMutexProfileFraction, runtime.SetBlockProfileRate) have their own costs and are off by default. A continuous profiler that collects them typically sets a low fraction, such as sampling 1 contention event in 100, and measures first.

What to collect

Traces are large and expensive, so they stay on demand (the tracer). The rest are cheap enough to collect all the time.

Labels and the value of the join

A raw profile says which functions used CPU. A labelled profile says on whose behalf. runtime/pprof supports labels on goroutines (pprof.Do(ctx, pprof.Labels("endpoint", "/search"), ...)) that are recorded with each sample, so a continuous profiler can answer "which endpoint costs the most CPU" and "how does tenant 42's CPU compare with the rest" without separate processes. Wrap the HTTP handler once with the route pattern as the label. Keep the label values bounded (route templates, not URLs; tenant tiers, not tenant ids), because every distinct value multiplies the storage.

The other join is version. Tag every profile with the build version and git commit, and the interesting query becomes a comparison: the flame graph of this release minus the last. Performance regressions then show up as boxes that grew, in the first hour of a rollout, before a customer notices. That is the main practical benefit of the whole exercise.

What it finds that a one-off profile does not

• Intermittent problems: the job that runs at 02:00, the cache refresh that spikes CPU every ten minutes.
• Slow regressions: a 2 percent increase per release that nobody notices until it is 30.
• Cross instance differences: one pod behaving differently from the others.
• Cost attribution: which functions account for the CPU bill, and what a refactor saved.
• Input for optimisation: the profiles that PGO consumes, collected from the real workload.

Fitting it into the other signals

Profiles answer a different question from metrics and traces, and a good setup uses all three.

• Metrics say that something is wrong: CPU is up, latency is up, memory is climbing. They are cheap and continuous, and they cannot say why.
• Traces say where one request spent its time across services. They follow a request, and they show waiting as clearly as computing.
• Profiles say where the CPU and memory of a process went, in aggregate, down to the line. They cannot follow a request, and they show only what consumed resources, not what waited.

A workable investigation goes metric, trace, profile: the dashboard shows the regression, the trace narrows it to one service and one operation, the profile shows the code. Linking them helps. Put the trace id into profile labels for the sampled requests (pprof labels are attached to goroutines, so you can set one per request), and put a link from the profile view to the dashboards. Pyroscope and similar tools support this "span profile" join, and the cost is a label on the handler.

A runbook for a regression

Here is the sequence continuous profiles make possible:

1. An alert says CPU per request rose 15 percent after yesterday's deploy.
2. Open the flame graph for the new version and diff it against the previous one. The boxes that grew are highlighted.
3. A box under encoding/json grew from 8 percent to 19 percent. Descend to the plateau; it is a new field that is marshalled with reflection on every request.
4. Check the allocation profile for the same two versions to see whether garbage grew too (allocation profiles).
5. Fix and write a benchmark for the path (benchmarking).
6. Deploy and confirm in the diff that the box shrank.

Without continuous profiling the same investigation starts by reproducing the load in a test environment, which for intermittent problems may never succeed.

Running it on Kubernetes and in containers

Two practical points. First, CPU limits matter: a container limited to a fraction of a core is throttled, and the profile reflects wall clock samples that include throttling, so a profile of a throttled service shows time in the scheduler and runtime.usleep that is not your code. Set GOMAXPROCS to the limit (or use a recent Go version, which does it from the container's CPU limit) so that the runtime and the profile agree with reality. Second, the profile must reach storage from an ephemeral pod: either push from inside the process, or run a pull based collector that scrapes the internal port on every pod it discovers. Both need the endpoint kept off the public network, which in a cluster means a ClusterIP service or a sidecar and a network policy.

Risks and how to manage them

Exposure. The profile endpoints reveal function names, file paths and internal structure, and heap and goroutine dumps can contain data. Serve them on an internal listener with authentication, or push from the process and never expose the port (securing pprof).

Collisions. Only one CPU profile can run at a time. If the agent holds it and an engineer requests /debug/pprof/profile, one of them gets an error ("cpu profiling already in use"). Use the agent's UI for ad hoc profiles, or pause the agent.

Data volume. A profile is tens of kilobytes. Per instance, once a minute, that is a few gigabytes a month per hundred instances, before compression and deduplication that the backends do. Set a retention policy.

Sensitive labels. Do not put user identifiers or request contents in labels. They end up in storage and in query results.

Overhead on tiny containers. On a 0.1 CPU limit the profiler's share is larger. Measure in that environment, as the lab did on its machine.

A rollout plan

1. Add the pprof endpoints on an internal port, protected, with labels for route and version.
2. Start with CPU and heap only, on a fraction of instances.
3. Measure the overhead on your service, the way the lab did: same load, with and without.
4. Roll out to all instances and keep the retention to days.
5. Add a habit: after each deploy, compare the new version's profile with the old.
6. When something regresses, the diff tells you where, and the allocation and CPU articles tell you how to fix it.

Checklist

• Capture short windows continuously; one CPU profile at a time per process.
• Keep CPU sampling at 100 Hz and MemProfileRate at its default.
• Tag profiles with version, instance and bounded labels.
• Compare versions after every deploy.
• Protect the endpoints and do not put identifiers in labels.
• Measure overhead on your own service before enabling it everywhere.

The runtime profilers are documented in the runtime/pprof package, and the general guidance in the Go diagnostics guide.