Heap Profiles in Go and How to Find What Holds Memory

Memory problems in Go are rarely about the language. They are about reachability: something still points at an object, so the garbage collector cannot free it. The heap profile is the tool for finding what is pointing. This article builds a program with a deliberate leak and a legitimate in-memory structure, takes two heap snapshots, and uses pprof's diff mode to find the leak in one command, with the numbers. Then it covers the parts of the heap profile that mislead: what the sampling does, why a snapshot needs a GC, and the difference between the four views.

It follows CPU profiling and reading flame graphs. The allocation side is the next article, leak hunting that is not goroutines is memory leaks in Go, and the production setup is net/http/pprof.

What the heap profile records

The runtime samples allocations. By default (runtime.MemProfileRate, 512 KB) it records the call stack of roughly one allocation per 512 KB allocated, and scales the counts up so that totals approximate the truth. For each stack it keeps four numbers:

inuse_* answers "what holds memory now". alloc_* answers "what allocates a lot" and is the one for reducing garbage (allocation profiles). A leak shows in inuse, because leaked memory is by definition live. A churn problem shows in alloc, because the objects are short lived and never accumulate.

Two details to remember. First, the profile reflects the heap as of the most recently completed garbage collection, so the live numbers lag behind the program. To get a current picture, call runtime.GC() before writing the profile, which is what the lab does; in a service, the handler for /debug/pprof/heap?gc=1 does the same. Second, sampling means small programs get noisy profiles, so the lab lowers the rate to 4096 bytes.

The lab

The program has two kinds of memory. sessions is legitimate: 100 sessions with 64 KB of state each, about 6.4 MB, bounded and needed. registry is the leak: every request stores a 4096 byte slice under a unique key and never removes it. Each request also creates a large temporary string that becomes garbage immediately. The program handles 20000 requests, snapshots, handles 40000 more, snapshots again:

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/pprof"
	"strings"
)

// registry is the leak: every request adds an entry and nothing removes it.
var registry = map[string][]byte{}

type session struct {
	id    string
	state []byte
}

// sessions is legitimate in-use memory: bounded and needed.
var sessions []*session

func handleRequest(i int) {
	registry[fmt.Sprintf("req-%d", i)] = make([]byte, 4096) // retained forever
	_ = strings.Repeat("temporary ", 1000)                  // garbage, freed by the GC
}

func openSession(i int) {
	sessions = append(sessions, &session{id: fmt.Sprint(i), state: make([]byte, 64<<10)})
}

func snapshot(name string) {
	runtime.GC() // a heap profile reports the state as of the last GC
	f, _ := os.Create(name)
	pprof.WriteHeapProfile(f)
	f.Close()
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Printf("%-7s HeapAlloc %6.1f MB, HeapObjects %7d\n", name, float64(m.HeapAlloc)/1e6, m.HeapObjects)
}

func main() {
	runtime.MemProfileRate = 4096 // sample about every 4 KB so a short program has stable numbers
	for i := 0; i < 100; i++ {
		openSession(i)
	}
	for i := 0; i < 20000; i++ {
		handleRequest(i)
	}
	snapshot("a.heap")
	for i := 20000; i < 60000; i++ {
		handleRequest(i)
	}
	snapshot("b.heap")
}

set -e
go build -o app .
./app
echo
echo "== inuse_space of the second snapshot, by function"
go tool pprof -sample_index=inuse_space -top -nodecount=5 app b.heap 2>/dev/null | grep -vE '^(Time|File|Build ID):'
echo
echo "== what grew between the snapshots (-base)"
go tool pprof -sample_index=inuse_space -base a.heap -top -nodecount=4 app b.heap 2>/dev/null | grep -vE '^(Time|File|Build ID):'
echo
echo "== inuse_objects of the second snapshot"
go tool pprof -sample_index=inuse_objects -top -nodecount=3 app b.heap 2>/dev/null | grep -vE '^(Time|File|Build ID):'
rm -f app a.heap b.heap

a.heap  HeapAlloc   91.8 MB, HeapObjects   40714
b.heap  HeapAlloc  260.9 MB, HeapObjects  121097

== inuse_space of the second snapshot, by function
Type: inuse_space
Showing nodes accounting for 246.17MB, 99.70% of 246.91MB total
Dropped 22 nodes (cum <= 1.23MB)
      flat  flat%   sum%        cum   cum%
  239.90MB 97.16% 97.16%   240.61MB 97.45%  main.handleRequest
    6.26MB  2.54% 99.70%     6.26MB  2.54%  main.openSession (inline)
         0     0% 99.70%   246.87MB   100%  main.main
         0     0% 99.70%   246.87MB   100%  runtime.main

== what grew between the snapshots (-base)
Type: inuse_space
Showing nodes accounting for 160.07MB, 97.84% of 163.60MB total
Dropped 9 nodes (cum <= 0.82MB)
      flat  flat%   sum%        cum   cum%
  160.07MB 97.84% 97.84%   160.59MB 98.16%  main.handleRequest
         0     0% 97.84%   160.59MB 98.16%  main.main
         0     0% 97.84%   160.59MB 98.16%  runtime.main

== inuse_objects of the second snapshot
Type: inuse_objects
Showing nodes accounting for 114266, 99.51% of 114827 total
Dropped 22 nodes (cum <= 574)
Showing top 3 nodes out of 4
      flat  flat%   sum%        cum   cum%
     68097 59.30% 59.30%     114522 99.73%  main.handleRequest
     46169 40.21% 99.51%      46425 40.43%  fmt.Sprintf
         0     0% 99.51%     114797   100%  main.main

The two snapshots printed HeapAlloc from runtime.MemStats: 91.8 MB with 40714 objects, then 260.9 MB with 121097. The growth is 169 MB. The leak adds 40000 entries of 4 KB each (164 MB) plus map overhead and key strings. The sessions are constant. This is the shape of a leak in a metric: the line goes up with the traffic and does not come back down after a GC.

Finding it

The top view of the second snapshot. go tool pprof -sample_index=inuse_space -top printed a header of Showing nodes accounting for 246.17MB, 99.70% of 246.91MB total and the table:

• main.handleRequest: 239.90 MB flat, 97.16 percent.
• main.openSession: 6.26 MB, 2.54 percent.
• main.main and runtime.main with 0 flat and the full cum, as callers.

One function holds 97 percent of live memory. The profile did not need to be clever: when a leak is large, -top finds it. Remember the limitation: the profile names where the memory was allocated, not who holds the reference. It says handleRequest allocated the 4 KB slices. That the global registry map keeps them reachable is the part you work out by reading handleRequest. In larger programs the allocating function is often a generic helper, and the useful clue is the caller chain (-peek, -tree, or the flame graph).

The diff. A single snapshot cannot tell leaked memory from memory that is supposed to be there. The sessions are 6 MB of live data, and a profile of a healthy service looks much like a leaking one at a glance. The diff separates them. go tool pprof -base a.heap b.heap subtracts the first snapshot from the second, so the output is only what changed:

Showing nodes accounting for 160.07MB, 97.84% of 163.60MB total

main.handleRequest 160.07 MB flat. Everything else disappears, including openSession, which did not grow. The diff is the leak hunter's best tool: take a snapshot after the service has warmed up, wait (for an hour, or for a thousand requests), take another and diff. If a function's inuse_space grows between them, you have found a candidate. This works in production: fetch /debug/pprof/heap?gc=1 twice and pass both files to pprof.

The object view. inuse_objects of the second snapshot reads: handleRequest 68097 objects (59 percent) and fmt.Sprintf 46169 (40 percent). The count is higher than the 40000 slices because the profile includes the first 20000 requests plus extrapolation, and the map keys: each key is a string built by fmt.Sprintf, also retained. The object view finds problems that space hides, such as a million tiny objects (map entries, small structs) that add up to little memory per object and a lot of garbage collector work, because the collector's cost scales with the number of live pointers.

The garbage that does not appear. Each request created a 10000 character temporary string (strings.Repeat("temporary ", 1000)). Across 60000 requests that is 600 MB of allocation. None of it shows in inuse_space, because the collector freed it. It would show in alloc_space. A reader who looked only at the allocation view would blame the temporary string and miss the leak; a reader who looked only at the in-use view would miss the garbage that costs CPU. Look at both, and know which question each answers.

Reading the numbers against the runtime

HeapAlloc (from runtime.MemStats) and the profile's total will not match exactly. The profile is sampled and extrapolated, and it describes the heap at the last GC. The process's resident set (what the operating system reports) is larger than both, because the runtime keeps freed memory around for reuse and returns it to the OS lazily (RSS against the heap). A graph of RSS going up does not prove a leak in your code. The heap profile does.

A leak checklist

When inuse_space grows without bound, the usual holders are:

• A package level map, slice or cache that only grows. The lab's registry. The fix is a size limit, an eviction policy or a TTL.
• A slice that keeps a large backing array alive after you take a small piece of it (slices that hold arrays).
• A goroutine that never exits and holds references. Look at the goroutine profile too (goroutine leaks).
• A timer, ticker or context that is never stopped and keeps its closure alive.
• Something registered and never unregistered: an observer, a hook, a metric label with unbounded cardinality.

For each, the heap profile gives you the allocating function and the diff tells you whether it is growing. Then read the code that retains the references.

Doing it in production

The net/http/pprof handler serves /debug/pprof/heap. A good routine: fetch the profile with ?gc=1 at a quiet moment, fetch another after some traffic, and diff them locally. The cost is small, and the default sampling rate is chosen so that the overhead is negligible. Do not set MemProfileRate to 1 in production, which records every allocation and slows the program. Do protect the endpoint (securing it), because the profile names every function in your binary.

Checklist

• Call runtime.GC() (or use ?gc=1) before reading a heap profile.
• Use inuse_space for what holds memory and alloc_space for what allocates.
• Take two snapshots and diff with -base; the function that grew is the candidate.
• Remember that the profile names the allocator, and you must find the retainer.
• Compare inuse_objects as well as space when the object count is the issue.
• Keep MemProfileRate at its default in production.

The profile format is documented in the runtime/pprof package and the diagnostics overview in the Go diagnostics guide.