Fix Memory Leaks in Go That Are Not Goroutine Leaks

When people say "Go memory leak" they usually mean a goroutine leak, because goroutines are the leak the language is famous for: a goroutine blocked forever on a channel pins its stack and everything it references. That is one source. The garbage collector frees only memory that nothing reaches, and a program can keep memory reachable in many other ways without starting a single goroutine. The symptoms are the same (live heap climbing, a container killed for exceeding its limit, a graph that never comes back down after traffic falls) and the goroutine count is flat, which sends people looking in the wrong place. This article builds four such leaks, measures how much each retains after 100000 iterations, shows the one line fix for each, and describes how to find a leak of this kind in a running service.

It pairs with heap profiles, which is the tool for locating the retaining type, and slices that keep huge arrays alive and maps that never shrink, which cover two of the usual suspects in more depth. When the number you are watching is RSS rather than live heap, read RSS against the heap first. The cornerstone is the Go performance guide.

What a leak is in a garbage collected language

A leak in Go is memory that is reachable from a root and never used again. The roots are package level variables, the stacks of running goroutines, and registers. Anything reachable from a root is live by definition, so the collector keeps it, however long ago the program stopped caring. So the question when hunting a leak is not "who forgot to free this" but "who still points at this". The four cases below are four different kinds of pointer: a global container, a hidden reference inside a value, a registry owned by a library, and a captured variable.

The experiment

Each scenario runs twice: once in the leaky form and once fixed. After each run the program forces two collections and reports the growth of HeapAlloc over a baseline in kilobytes. The baseline was 149 KB. All four scenarios do 100000 iterations.

package main

import (
	"context"
	"fmt"
	"runtime"
	"strings"
)

func live() uint64 {
	runtime.GC()
	runtime.GC()
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	return m.HeapAlloc
}

type scenario struct {
	name  string
	leaky func(n int) func()
	fixed func(n int) func()
}

var sink any

// 1. a package level map used as a cache with no bound.
var cache = map[int][]byte{}

func cacheLeaky(n int) func() {
	for i := 0; i < n; i++ {
		cache[i] = make([]byte, 1024)
	}
	return func() { cache = map[int][]byte{} }
}

func cacheFixed(n int) func() {
	const max = 1000
	local := map[int][]byte{}
	for i := 0; i < n; i++ {
		if len(local) >= max {
			for k := range local {
				delete(local, k)
				break
			}
		}
		local[i] = make([]byte, 1024)
	}
	sink = local
	return func() { sink = nil }
}

// 2. a short substring that keeps its large parent string alive.
var tokens []string

func substringLeaky(n int) func() {
	for i := 0; i < n; i++ {
		big := strings.Repeat("x", 4096) + "TOKEN"
		tokens = append(tokens, big[len(big)-5:])
	}
	return func() { tokens = nil }
}

func substringFixed(n int) func() {
	for i := 0; i < n; i++ {
		big := strings.Repeat("x", 4096) + "TOKEN"
		tokens = append(tokens, strings.Clone(big[len(big)-5:]))
	}
	return func() { tokens = nil }
}

// 3. cancellable child contexts that nobody cancels stay registered on the parent.
var parent, parentCancel = context.WithCancel(context.Background())

func ctxLeaky(n int) func() {
	for i := 0; i < n; i++ {
		ctx, _ := context.WithCancel(parent)
		_ = ctx
	}
	return func() { parentCancel(); parent, parentCancel = context.WithCancel(context.Background()) }
}

func ctxFixed(n int) func() {
	for i := 0; i < n; i++ {
		ctx, cancel := context.WithCancel(parent)
		_ = ctx
		cancel()
	}
	return func() {}
}

// 4. a callback list whose closures capture a large buffer they never use.
var handlers []func()

func closureLeaky(n int) func() {
	for i := 0; i < n; i++ {
		buf := make([]byte, 4096)
		id := i
		handlers = append(handlers, func() { _ = id; _ = len(buf) })
	}
	return func() { handlers = nil }
}

func closureFixed(n int) func() {
	for i := 0; i < n; i++ {
		buf := make([]byte, 4096)
		size := len(buf)
		id := i
		handlers = append(handlers, func() { _ = id; _ = size })
	}
	return func() { handlers = nil }
}

func main() {
	const n = 100000
	scenarios := []scenario{
		{"unbounded map cache, 1 KB values", cacheLeaky, cacheFixed},
		{"5 byte substring of a 4 KB string", substringLeaky, substringFixed},
		{"child contexts never cancelled", ctxLeaky, ctxFixed},
		{"closures capturing a 4 KB buffer", closureLeaky, closureFixed},
	}
	base := live()
	fmt.Printf("baseline live heap: %d KB, %d iterations per scenario\n\n", base>>10, n)
	fmt.Printf("%-36s %14s %14s\n", "scenario", "leaky (KB)", "fixed (KB)")
	for _, s := range scenarios {
		undo := s.leaky(n)
		l := live() - base
		undo()
		live()
		undo = s.fixed(n)
		f := live() - base
		undo()
		live()
		fmt.Printf("%-36s %14d %14d\n", s.name, l>>10, f>>10)
	}
}

baseline live heap: 149 KB, 100000 iterations per scenario

scenario                                 leaky (KB)     fixed (KB)
unbounded map cache, 1 KB values             105155           1151
5 byte substring of a 4 KB string            476871           2406
child contexts never cancelled                11266             41
closures capturing a 4 KB buffer             405665           3321

Results

None of these programs started a goroutine, and each leak was caused by a pointer from something that lives for the whole run into something that should have been short lived.

Leak 1: the cache with no bound

A package level map[int][]byte is the simplest leak there is and the one most often written in good faith. A cache is a map you add to. Nothing in the language removes an entry, so unless your code evicts, the map is a one way valve. In the lab 100000 values of 1 KB retained 105,155 KB, about 103 MB, which is the 100 MB of payload plus map overhead. The fix in the lab caps the map at 1000 entries and evicts an arbitrary one when full, which retained 1,151 KB.

The honest fix in a real service is a cache with a policy: a least recently used list with a maximum size, a time to live, or both. Pick the bound from memory, not from entry count, if values vary in size. A cache without a bound is a design decision to hold every key the service has ever seen, and key spaces driven by user input (session IDs, URLs, query strings) are effectively unbounded. The same applies to a map of per user state, a set of seen request IDs used for deduplication, and a registry of metrics labels keyed by an attacker controlled value.

How to see it: a heap profile with inuse_space shows the map's allocation site at the top, and the profile grows in proportion to the number of distinct keys, not to the request rate.

Leak 2: the small substring that keeps a big string

A Go string is a pointer and a length. Slicing a string gives a new string that points into the same backing bytes. That makes slicing free, and it also means a 5 byte token cut from a 4 KB line keeps all 4 KB alive for as long as the token lives. In the lab each iteration built a string of 4,101 bytes, kept the last five, and discarded the rest, so 100000 iterations retained 476,871 KB, about 466 MB, to hold 500 KB of tokens. The fix is strings.Clone, which copies the five bytes into their own allocation, and the retained heap fell to 2,406 KB.

The same happens with []byte sub slices and with the results of bufio.Scanner.Text() on code that retains Bytes(), with fields split out of a large request body with strings.Split, and with values parsed from a large JSON document into string fields by a decoder that slices the input. The pattern to look for is a long lived structure that stores a short piece of a large input. Clone at the boundary where you decide to keep it, and not before, since cloning everything defeats the point of slices. Go 1.20 added strings.Clone for exactly this; before that, string([]byte(s)) was used, which also copies.

Leak 3: contexts that nobody cancels

context.WithCancel(parent) registers the child with its parent so that cancelling the parent cancels the child. The parent holds a map of its children, and an entry is removed only when the child is cancelled. A child whose cancel is never called therefore stays in the parent's map for as long as the parent lives. In a request handler this is harmless because the request context ends and takes its children with it, which is why the mistake survives code review. In a long lived process context, such as one created in main and handed to a worker loop that makes a child per item, it is a leak. In the lab 100000 uncancelled children retained 11,266 KB, about 110 bytes each, and calling cancel brought the heap back to 41 KB.

The language tooling catches this one: go vet reports a lost cancel function ("the cancel function is not used on all paths"). The fix is always defer cancel() immediately after the call, even when you also pass a timeout, because the timer in WithTimeout is released on cancel instead of at expiry. A leak of timers behaves the same way: since Go 1.23 an unreferenced time.Timer or time.Ticker is collected without being stopped, which removed a classic leak, but one stored in a long lived struct is still referenced.

Leak 4: the closure that captures more than it uses

A closure captures variables, not values, and captures the whole variable. The leaky version in the lab allocated a 4 KB buffer, measured its length inside the closure, and stored the closure in a global list. The closure keeps the buffer alive because it mentions buf. 100000 of them retained 405,665 KB, about 396 MB. The fix copies what the closure needs (size := len(buf)) before creating it, so the closure holds an int and the buffer is garbage when the function returns, and the list retained 3,321 KB.

This is the most common leak in callback driven code: event handlers, middleware chains, deferred functions that capture a request, sync.Once functions, time.AfterFunc callbacks, subscriber lists that are registered once per connection and never removed. The closure looks small and carries an entire request. If a long lived list holds functions, check what each function captures. A method value (obj.Method) captures the receiver obj, which is a quieter version of the same thing.

How to find one in a running service

Goroutine count tells you nothing for these leaks, so use the heap.

1. Confirm a leak, not a high water mark. Plot the live heap after collection (/gc/heap/live:bytes) over hours. A leak is a staircase that does not return to the previous floor after traffic drops. A high water mark returns (RSS against the heap).
2. Take two heap profiles an hour apart from the same instance and diff them: go tool pprof -base first.pb.gz second.pb.gz. The top of the diff by inuse_space is the growing allocation site (heap profiles).
3. Read the allocation site as a clue, not as the culprit. The profile shows where the retained memory was allocated. The retaining pointer is somewhere else. For the substring leak the profile points at the code that built the big string; the fix is in the code that stored the token.
4. Ask who holds it. Look for the global variable, the long lived struct, the registry or the list that the allocation ends up in. Search the codebase for package level maps, slices and sync.Maps first, since they account for most of these.
5. Reproduce in a test. The lab's method works in a test: run the operation N times, call runtime.GC() twice, read HeapAlloc, and fail if it grew by more than a bound. A leak found once and pinned by a test stays fixed.

Pinning the fix with a test

A leak fix without a test is a fix that gets undone in a refactor. The lab's measuring function is the test: force two collections so that garbage is gone, read HeapAlloc, and compare with a bound. Written for one operation it looks like this:

func TestNoRetention(t *testing.T) {
	before := liveHeap()
	for i := 0; i < 100000; i++ {
		handle(sampleRequest)
	}
	grew := liveHeap() - before
	if grew > 4<<20 {
		t.Fatalf("live heap grew by %d bytes", grew)
	}
}

Choose the bound from a run of the fixed code with some slack, not from zero: the runtime allocates a little for itself, and a test that fails on a kilobyte of noise gets disabled. Run it without -race and without parallel tests, since both change allocation. A bound of a few megabytes catches every leak in the table above, because the smallest one retained 11 MB.

A short worked example

Suppose a service shows a live heap that climbs 40 MB an hour and the goroutine count is steady at 212. The heap profile's top entry by inuse_space is strings.Repeat in a function that builds log lines, which cannot be right: a log line is not kept. Following the allocation site up one frame shows the line is passed to an audit trail that stores line[start:end], a 12 byte request ID, in a package level slice used to reject duplicates. That is the substring leak. The fix is one call to strings.Clone where the ID is stored, and the next profile shows the allocation site gone. Two details of the process matter: the profile pointed at the wrong code (the allocator, not the holder), and the goroutine count never moved, so a dashboard built only on goroutines would have called the service healthy.

Other places to look

• sync.Pool that grows: a pool does not bound the size of what it holds (sync.Pool).
• Slices that are re-sliced but never trimmed: a queue implemented as q = q[1:] keeps the head of the array until it reallocates.
• http.Response bodies that are not closed: these leak connections and goroutines, so they are the goroutine kind, but the buffers show up in the heap too.
• Finalizers and runtime.SetFinalizer on objects that are referenced from a cycle that includes the finalizer's own closure.
• Logging and metrics libraries with unbounded label sets: a counter per user ID is a map entry per user.

Checklist

• A Go memory leak is a live pointer you did not expect, not a missing free.
• Look at package level maps, slices and registries before anything else.
• Clone a short piece of a big string or slice before keeping it.
• Call cancel on every derived context, and let go vet check it.
• Do not let a long lived closure capture a request, a buffer or a connection.
• Diagnose with two heap profiles and -base, since the goroutine count is flat.
• Pin the fix with a test that forces two collections and checks HeapAlloc.

The runtime behaviour is described in the Go GC guide, and strings.Clone and the context rules in the strings and context package documentation.