Tune the Go Garbage Collector With GOGC the Careful Way

GOGC is the Go garbage collector's one tuning knob, and it is widely misunderstood in both directions: people leave it alone forever on programs that would run twice as fast with a different value, and they change it to "fix pauses" that it does not affect. This article runs the same allocation heavy program under six GOGC values on Go 1.25, prints the number of collections, wall time, pause totals and memory, and ends with a procedure for deciding whether to touch it at all.

It is the first of three GC articles in the cluster: the memory limit that should accompany it is in GOMEMLIMIT, and measuring pauses is in GC pauses in production. The cheaper way to speed up a GC bound program is to allocate less (allocation profiles, reducing allocations). The cornerstone is the Go performance guide.

What GOGC does

Go's collector is concurrent and non moving. It marks reachable objects while the program runs, using spare CPU, and briefly stops the program at the start and end of a cycle. After a cycle that ends with L bytes live, the runtime sets the next trigger so that the heap can grow to L * (1 + GOGC/100) before collecting again.

So GOGC is a ratio of memory to CPU. A higher value lets the heap grow larger before the next cycle, which means fewer cycles, less CPU spent marking, and more memory used. A lower value does the opposite. A minimum heap of 4 MB times GOGC/100 applies so that tiny programs do not collect constantly.

Two things GOGC does not do. It does not shorten pauses: the stop-the-world phases of a cycle are short regardless (tens of microseconds, shown below). And it does not reduce the cost of a cycle: marking takes time proportional to the live heap's size and pointer count, however often it runs.

The experiment

The program builds a live heap of about 100 MB (1.6 million 64 byte nodes linked in a list: small, pointer rich objects, which are expensive to mark), forces a collection, then allocates 600000 short lived 4 KB buffers, a stand in for request handling that allocates and discards. It reports wall time for the churn phase, the number of collections during it, pause totals, the GC CPU fraction and HeapSys, then exits. A script runs it six times with different GOGC values:

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/debug"
	"time"
)

type node struct {
	next *node
	val  [6]int64
}

var keep [][]byte

// build makes a live heap of about mb megabytes of small pointer-rich nodes.
func build(mb int) *node {
	var head *node
	for i := 0; i < mb*1024*1024/64; i++ {
		head = &node{next: head}
		head.val[0] = int64(i)
	}
	return head
}

func main() {
	live := 100
	if len(os.Args) > 1 {
		fmt.Sscan(os.Args[1], &live)
	}
	heap := build(live)
	runtime.GC()
	var before runtime.MemStats
	runtime.ReadMemStats(&before)

	start := time.Now()
	// Churn: short lived 4 KB buffers, the shape of request handling.
	var sum int
	for i := 0; i < 600000; i++ {
		b := make([]byte, 4096)
		b[0] = byte(i)
		sum += int(b[0])
		if i%1000 == 0 {
			keep = append(keep[:0], b)
		}
	}
	elapsed := time.Since(start)

	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	gcs := m.NumGC - before.NumGC
	pause := time.Duration(m.PauseTotalNs - before.PauseTotalNs)
	limit := debug.SetMemoryLimit(-1)
	lim := "none"
	if limit < 1<<62 {
		lim = fmt.Sprintf("%d MiB", limit>>20)
	}
	fmt.Printf("GOGC=%-4s limit=%-8s %5d ms  %4d GCs  pauses %4d µs total  GC CPU %4.1f%%  peak heap %4d MB\n",
		os.Getenv("GOGC"), lim, elapsed.Milliseconds(), gcs, pause.Microseconds(), m.GCCPUFraction*100, m.HeapSys>>20)
	_ = sum
	_ = heap.val[0]
}

set -e
go build -o app .
echo "live heap about 100 MB of 64 byte nodes, then 600000 short lived 4 KB buffers"
for g in 25 50 100 200 400 800; do GOGC=$g ./app 100; done
rm -f app

live heap about 100 MB of 64 byte nodes, then 600000 short lived 4 KB buffers
GOGC=25   limit=none      3035 ms    28 GCs  pauses  420 µs total  GC CPU  9.6%  peak heap  899 MB
GOGC=50   limit=none      3022 ms    28 GCs  pauses  384 µs total  GC CPU  9.6%  peak heap  771 MB
GOGC=100  limit=none      2064 ms    18 GCs  pauses  276 µs total  GC CPU  9.3%  peak heap  271 MB
GOGC=200  limit=none      1007 ms     8 GCs  pauses  161 µs total  GC CPU  8.4%  peak heap  459 MB
GOGC=400  limit=none       500 ms     3 GCs  pauses   61 µs total  GC CPU  6.9%  peak heap  871 MB
GOGC=800  limit=none       524 ms     1 GCs  pauses   16 µs total  GC CPU  6.4%  peak heap 1755 MB

Results

Time follows the number of collections

From 28 collections to 3 the run went from 3035 ms to 500 ms. Dividing, each collection cost about 90 ms: (3035 − 500) / 28. That is the price of marking 100 MB made of 1.6 million small objects with pointers, during which the mutator (the program's own goroutines) is also slowed, since a goroutine that allocates while a cycle runs is made to help with the marking (an assist). The churn phase on its own, with almost no collection, takes about 500 ms: 600000 allocations of 4 KB is 2.4 GB of zeroed memory.

So for this program, GOGC is a 6 times lever on run time. That is unusual: it comes from a large live heap that is expensive to mark combined with a high allocation rate. Programs with small live heaps barely notice GOGC, since a cycle takes microseconds.

Pauses did not change

The pause column is the sum of all stop-the-world pauses in the phase: 420 µs for 28 collections at GOGC=25, 276 µs for 18 at the default, 16 µs for 1 at 800. Each pause is 10 to 25 µs, and they are the same size whatever the setting. The total is below half a millisecond in a run of three seconds. If a service has latency spikes, they come from elsewhere (assists slowing request goroutines, scheduling, locks, the network) and not from stop-the-world pauses; changing GOGC to "fix pauses" is a common mistake. The pause article measures this further.

Memory is the price

At GOGC=400 the peak heap was 871 MB, at 800 it was 1755 MB, against 271 MB at the default. Higher GOGC buys speed with memory, as expected. Whether you can spend it depends on where the program runs: a container with a 512 MB limit cannot, a machine with 64 GB and one process can.

Lower GOGC did not save memory here

At GOGC=25 and 50 the peak was larger than at 100 (899 MB and 771 MB against 271 MB), which looks wrong. The explanation is concurrency. The collector marks while the program keeps allocating, and this program allocates 2.4 GB in about half a second of mutator time, much faster than a cycle completes (90 ms). During one cycle the program allocates hundreds of megabytes more, and the heap overshoots the trigger. More frequent cycles mean the program spends more of its time in cycles, and the heap is often mid cycle, growing. For real services with allocation rates in the hundreds of megabytes per second, rather than 5 GB/s, the overshoot is smaller, and lowering GOGC does reduce the peak, at a CPU cost. The lesson holds: peak memory is the live heap plus what is allocated during a cycle plus the headroom GOGC allows, and the lab's allocation rate dominates the second term.

A procedure

1. Check that GC is a problem at all. A CPU profile with runtime.gcBgMarkWorker, runtime.mallocgc and assists at more than 10 to 20 percent of samples (the CPU article), or GODEBUG=gctrace=1 showing a GC share above 10 percent. If the numbers are small, stop.
2. Reduce allocation first. A smaller allocation rate means fewer cycles whatever GOGC says, and it is usually the biggest lever (allocation profiles).
3. Set a memory budget (the container limit minus headroom) and GOMEMLIMIT to a fraction of it (GOMEMLIMIT).
4. Raise GOGC while the heap stays inside the budget. With a live heap of 100 MB and a 400 MB budget, GOGC around 200 to 300 gives the program room. Watch the GC CPU share and the peak RSS (RSS and the heap).
5. Alternatively, use GOGC=off with GOMEMLIMIT for a process that owns its memory: the collector runs only when the heap approaches the limit. It is a good pattern for a batch job or a single tenant container, and risky if the live heap can approach the limit, because the collector then runs back to back.
6. Re-measure with the real workload. The lab is synthetic. The shape of the curve (time falls as GOGC rises, memory rises) is general; the numbers are not.

You can set GOGC with the environment variable or at run time with debug.SetGCPercent, which returns the old value and takes effect for the next cycle. Setting it from configuration in main lets you change it without rebuilding.

Picking a number from a memory budget

You can turn the procedure into arithmetic. If the live heap after a cycle is L and the memory you are willing to spend on the Go heap is B, the largest GOGC that stays inside the budget is about (B / L - 1) * 100, ignoring overshoot. With the lab's L of 100 MB:

The table is optimistic for two reasons. The live heap changes (it is a sawtooth, not a constant), so take L from the largest value you see in gctrace over a representative day. And the heap overshoots the trigger by the amount the program allocates during a cycle, which for the lab was several hundred megabytes. Leave a margin of 20 to 30 percent of the budget, and let GOMEMLIMIT catch the rest: it makes the collector run harder as the heap approaches the limit, whatever GOGC says, which is its purpose (the memory limit article).

For a service whose live heap is small, say 20 MB, GOGC 100 means a trigger at 40 MB, and cycles are quick. The 4 MB minimum heap times GOGC/100 matters only for tiny programs. In this regime the interesting number is the cycle frequency (how often per second), and a high allocation rate with a small heap means very frequent cheap cycles. If that frequency is above a few per second and the GC share is high, raising GOGC to 200 or 400 costs little memory because the live heap is small: 20 MB at GOGC 400 is a 100 MB trigger. This is the most profitable case for tuning: small live heap, high allocation rate.

The opposite case is a large live heap (several gigabytes) with a modest allocation rate. Cycles are expensive but rare. Raising GOGC there multiplies the memory (a 4 GB live heap at GOGC 200 can reach 12 GB), so the trade is usually bad, and lowering allocation or splitting the heap across processes is better.

GOGC, GOMAXPROCS and containers

The collector uses about a quarter of the available processors for background marking, and slows the program further with assists when allocation outpaces marking. On a machine with 10 cores that is a couple of cores of spare capacity; in a container limited to one CPU, marking competes directly with the application and every cycle visibly lengthens request latency. Two practical rules follow:

• Set GOMAXPROCS to the container's CPU limit. Recent Go releases pick it from the cgroup limit, and older ones need a helper package or the environment variable. With the wrong value the scheduler plans for cores you do not have.
• Tune GOGC under the same CPU limit you run in production. A test on a ten core laptop shows the collector hidden by idle cores, and the same program on one core pays for every cycle in latency.

The scavenger, which returns unused memory to the operating system in the background, is a separate mechanism with its own pace; it explains why resident memory lags behind the heap after a spike (RSS against the heap).

What to monitor after changing it

A change to GOGC is a change to the service's resource profile, so watch it for a week, not an hour:

• Resident memory and the Go heap (go_memstats_heap_inuse_bytes if you export runtime metrics) at peak traffic. The new ceiling should sit inside the budget.
• GC CPU share (/cpu/classes/gc/total:cpu-seconds in runtime/metrics, or GCCPUFraction). It should fall.
• Request latency percentiles. Fewer assists should lower p99; a bigger heap can slow it if the working set stops fitting in cache or if the operating system starts swapping.
• Out of memory kills and restarts. Any is a failed experiment.

Keep the old value one config change away.

Reading gctrace

GODEBUG=gctrace=1 prints a line per cycle to standard error. The lab shows the first, with times masked: gc 1 @T P: A+B+C ms clock, ... ms cpu, 3->4->4 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 10 P. The fields to read: gc N the cycle number; @T P the time since start and the percentage of CPU spent in GC so far; A+B+C ms clock the stop-the-world sweep termination, the concurrent mark and the mark termination; 3->4->4 MB the heap size at the start, at the end and the live size after marking; 4 MB goal the target; 10 P the number of processors. The P percentage is the number to watch: sustained values above 10 to 25 percent are the sign that GC tuning or an allocation fix would pay.

Mistakes

• Turning GOGC up without a memory limit, then being killed by the out of memory handler under load.
• Turning it down to "save memory" on a program whose allocation rate makes the heap overshoot anyway.
• Setting it to fix tail latency caused by something else.
• Tuning on a laptop for a service that runs with 1 CPU in a container: with few cores the collector competes with the program for the same CPUs, and the trade changes.

Checklist

• GOGC is a memory to CPU ratio; it does not change pause length.
• The cost of a cycle is proportional to the live pointer rich heap.
• Measure the GC share with a profile or gctrace before touching it.
• Cut allocations before tuning; add GOMEMLIMIT as a guard.
• Raise GOGC only inside a stated memory budget, and re-measure.

The collector is described in the Go GC guide and the knobs in the runtime package documentation.