GOMEMLIMIT in Go and Running Close to a Memory Cap

Before Go 1.19 the only way to tell the garbage collector about a memory budget was GOGC, which is a ratio and knows nothing about the machine. A service in a container with a 512 MiB limit had to pick a GOGC that kept the worst case under it, wasting memory the rest of the time, or risk the kernel killing it for exceeding the limit. GOMEMLIMIT, added in Go 1.19, is the missing half: a soft limit on the total memory the Go runtime will use. The collector works as little as it can below the limit and harder as the total approaches it. This article runs the same allocation heavy program under limits from 800 MiB down to 100 MiB, prints the collections and times, and describes how to set a limit that protects a container without choking it.

It follows GOGC tuning, whose experiment and workload it reuses, and leads to GC pauses and RSS against the heap. The cornerstone is the Go performance guide.

What the limit counts

GOMEMLIMIT (or debug.SetMemoryLimit(bytes)) bounds the memory the Go runtime manages: the heap, goroutine stacks, runtime internal structures, and memory the runtime holds but has not returned to the operating system. It does not count memory outside the runtime's control: cgo allocations, memory mapped files, the binary itself, and anything the kernel accounts to the process that the runtime does not know about. In a container, set the limit below the container's limit by a margin for those (10 percent is a common start).

The value accepts suffixes: GOMEMLIMIT=450MiB. The default is effectively unlimited. It interacts with GOGC like this:

• With the default GOGC=100 and a limit, the collector runs when either condition is met: the heap has doubled since the last cycle, or total memory is near the limit. The limit rarely triggers if GOGC would already collect sooner.
• With GOGC=off and a limit, the collector runs only when memory nears the limit. The heap grows freely up to it and no further, so a process uses as much memory as it is given and collects as rarely as possible. This is the configuration for a process that owns its memory budget.

The GC CPU limiter

A hard memory limit has a famous failure mode: if the live data is bigger than the limit, a collector that must stay under it runs continuously and the program makes no progress (a "death spiral"). The Go runtime guards against it. A GC CPU limiter caps the fraction of CPU the collector may use (50 percent in the current design), so when the limit cannot be met the program slows down but keeps running, and the memory limit is exceeded softly. That is why the name says limit and the documentation says soft. A limit is a target for the collector, not an enforced ceiling.

The experiment

The same program as in the GOGC article: a live heap of about 100 MB of small pointer rich nodes, then 600000 short lived 4 KB buffers. The script runs it once with the default settings and then with GOGC=off and each limit in turn, then with the limits and the default GOGC together:

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/debug"
	"time"
)

type node struct {
	next *node
	val  [6]int64
}

var keep [][]byte

// build makes a live heap of about mb megabytes of small pointer-rich nodes.
func build(mb int) *node {
	var head *node
	for i := 0; i < mb*1024*1024/64; i++ {
		head = &node{next: head}
		head.val[0] = int64(i)
	}
	return head
}

func main() {
	live := 100
	if len(os.Args) > 1 {
		fmt.Sscan(os.Args[1], &live)
	}
	heap := build(live)
	runtime.GC()
	var before runtime.MemStats
	runtime.ReadMemStats(&before)

	start := time.Now()
	// Churn: short lived 4 KB buffers, the shape of request handling.
	var sum int
	for i := 0; i < 600000; i++ {
		b := make([]byte, 4096)
		b[0] = byte(i)
		sum += int(b[0])
		if i%1000 == 0 {
			keep = append(keep[:0], b)
		}
	}
	elapsed := time.Since(start)

	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	gcs := m.NumGC - before.NumGC
	pause := time.Duration(m.PauseTotalNs - before.PauseTotalNs)
	limit := debug.SetMemoryLimit(-1)
	lim := "none"
	if limit < 1<<62 {
		lim = fmt.Sprintf("%d MiB", limit>>20)
	}
	fmt.Printf("GOGC=%-4s limit=%-8s %5d ms  %4d GCs  pauses %4d µs total  GC CPU %4.1f%%  peak heap %4d MB\n",
		os.Getenv("GOGC"), lim, elapsed.Milliseconds(), gcs, pause.Microseconds(), m.GCCPUFraction*100, m.HeapSys>>20)
	_ = sum
	_ = heap.val[0]
}

set -e
go build -o app .
echo "live heap about 100 MB, then 600000 short lived 4 KB buffers; GOGC=off means only the memory limit triggers the collector"
GOGC=100 ./app 100
for lim in 800MiB 300MiB 200MiB 150MiB 120MiB 100MiB; do GOGC=off GOMEMLIMIT=$lim ./app 100; done
echo
echo "the same limits with the default GOGC=100 also in force"
for lim in 150MiB 120MiB; do GOGC=100 GOMEMLIMIT=$lim ./app 100; done
rm -f app

live heap about 100 MB, then 600000 short lived 4 KB buffers; GOGC=off means only the memory limit triggers the collector
GOGC=100  limit=none      1897 ms    16 GCs  pauses  237 µs total  GC CPU  9.2%  peak heap  291 MB
GOGC=off  limit=800 MiB    517 ms     3 GCs  pauses   63 µs total  GC CPU  6.6%  peak heap  755 MB
GOGC=off  limit=300 MiB   1577 ms    13 GCs  pauses  441 µs total  GC CPU  8.9%  peak heap  283 MB
GOGC=off  limit=200 MiB   2833 ms    25 GCs  pauses  731 µs total  GC CPU  9.4%  peak heap  475 MB
GOGC=off  limit=150 MiB   3045 ms    29 GCs  pauses 2248 µs total  GC CPU  9.6%  peak heap 1019 MB
GOGC=off  limit=120 MiB   3120 ms    34 GCs  pauses 1059 µs total  GC CPU  9.5%  peak heap 1023 MB
GOGC=off  limit=100 MiB   3214 ms    31 GCs  pauses  603 µs total  GC CPU  9.7%  peak heap  931 MB

the same limits with the default GOGC=100 also in force
GOGC=100  limit=150 MiB   3168 ms    30 GCs  pauses  827 µs total  GC CPU  9.7%  peak heap  743 MB
GOGC=100  limit=120 MiB   2987 ms    30 GCs  pauses  770 µs total  GC CPU  9.6%  peak heap  899 MB

Results

A generous limit is the fastest setting

With 800 MiB available and the collector turned off, the program ran in 517 ms, nearly four times faster than the default, with only 3 collections: the heap grew to what the limit allowed and the collector ran just three times, when the limit was approached. It is the same effect as a very high GOGC in the previous article, with a safety net. Nothing was lost to GC because there was room.

Tightening the limit trades time for memory

As the limit fell the collections rose and the run slowed: 13 collections and 1577 ms at 300 MiB, 25 and 2833 ms at 200 MiB, 29 and 3045 ms at 150 MiB. That is the intended behaviour. The collector works harder to keep the process under a budget, and the cost shows as time. Each collection costs about 90 ms here (a 100 MB heap of 1.6 million small objects), so 29 collections cost about 2.6 seconds.

Below the live heap, it still finishes

At 120 MiB and 100 MiB the limit is at or below the live heap plus the runtime's own overhead. A collector that tried to honour it exactly could never succeed. The program still completed, in 3120 and 3214 ms, with 34 and 31 collections: the same order as at 150 MiB. Without the GC CPU limiter this configuration would run the collector back to back and make very little progress. With it, the cost is bounded: the program is about 6 times slower than the unconstrained one, and no more. This is a result you can use: the downside of setting the limit too low is slowness, not a hang, though 6 times slower is still an incident.

Peak heap is not a hard cap

HeapSys (virtual memory obtained from the OS for the heap) reached 1019 MB at a limit of 150 MiB. The limit is soft, and this program allocates 4 KB buffers at several gigabytes per second, faster than a 90 ms collection cycle finishes. During a cycle the heap grows well past where the collector wanted it. HeapSys also counts memory the runtime has not yet given back to the OS, so it overstates what is in use; the resident set size is what the kernel's out of memory handler looks at, and it depends on how quickly the runtime returns memory (RSS against the heap). For workloads that allocate more slowly, the heap tracks the limit closely.

Adding GOGC changes little

The last two rows (GOGC=100 with limits of 150 and 120 MiB) look like the GOGC=off rows with the same limits: 30 collections and about 3 seconds. When the limit is tight, it is the limit that decides when the collector runs. Leaving GOGC at 100 is the safer choice for production because it means a service with a large limit but a small heap still collects at a reasonable rate, instead of letting memory grow to the limit before the first cycle.

Setting a limit in a container

A recipe:

1. Know the container limit (say 1 GiB) and the live heap you see at peak (say 300 MiB).
2. Set GOMEMLIMIT to about 90 percent of the container limit: 900 MiB, leaving room for stacks you did not predict, cgo, mapped files and the kernel's accounting.
3. Keep GOGC at its default unless the profile shows GC taking more than 10 to 20 percent of CPU, then raise it within the budget (GOGC).
4. Use GOGC=off only for processes where memory is yours to spend and the live heap is much smaller than the limit, such as a batch job or a single tenant worker. If the live heap can grow towards the limit, the collector will run constantly.
5. Watch GC CPU share and restarts. If the limiter is active (the metric /gc/limiter/last-enabled:gc-cycle in runtime/metrics shows recent activity), the limit is too tight for the live heap: raise the container limit or reduce the live heap.

With debug.SetMemoryLimit you can read the current value (pass -1) and change it at run time, for example from a configuration reload or to leave room for a known large allocation.

What a limit does not do

• It does not prevent out of memory kills by itself. A leak that grows the live heap past the container limit will still be killed, only later and after the program has spent much of its CPU on collections. A rising GC share is a leak symptom (heap profiles).
• It does not cover non Go memory. A program that mmaps a big file or uses cgo needs a bigger margin.
• It does not shorten pauses. They were 63 to 2248 µs in total across the runs.
• It does not replace reducing allocation. At every limit in the table, a program that allocated less would have run faster.

Reading the numbers as a cost per collection

The table hides a simple model. Each collection of this heap costs roughly the same, because the live data is fixed at about 100 MB of small pointer rich nodes and the collector must mark all of it every cycle. At the default settings that was 16 cycles in 1897 ms. At 150 MiB it was 29 cycles in 3045 ms. The difference, 13 extra cycles and about 1150 extra milliseconds, gives roughly 90 ms per additional cycle, which matches the mark time you see in a trace of this heap. So a limit does not make collection slower, it makes it more frequent, and the bill is the number of cycles times the cost of marking what is live.

That is why the limit hurts programs with a large live heap and barely touches programs with a small one. A service with 20 MB of live data and a 1 GiB limit would run a cycle in a few milliseconds even if the limit forced hundreds of them. A service with 2 GB of live data and a 2.2 GB limit would spend most of its time marking. The ratio that matters is live heap to limit, the same ratio GOGC expresses, only now fixed by a number of bytes instead of a percentage.

Choosing the headroom

The headroom is the gap between the live heap and the limit. The collector can only allocate into that gap between cycles, so a small gap means short cycles back to back. A rule that fits the measurements above:

• With headroom of 8 times the live heap (800 MiB for 100 MB), the collector ran 3 times.
• With 3 times (300 MiB), 13 times.
• With 1.5 times (150 MiB), 29 times.
• With 1 times or less, 31 to 34 times, where the CPU limiter flattened the curve.

For a service you want the normal operating point to sit at 2 times or more, so that a spike in live data has room before the collector turns aggressive. If a profile shows the live heap already at 70 percent of the container limit, the right fix is a bigger container or less retained data, because no setting of the runtime makes marking free.

Use it with an observed baseline

Before setting a limit on a real service, record three numbers over a busy day: the peak live heap after collection (/gc/heap/live:bytes in runtime/metrics), the peak resident set size from the container metrics, and the share of CPU spent in the collector. The first tells you the floor under any limit. The second tells you how much memory outside the heap the process uses, which is the margin you must leave. The third is the baseline you compare against after the change.

Then set the limit, deploy it to one instance and watch the same three numbers for a day. If the collector share stays within a point or two of the baseline and the resident set stops brushing the container limit, the change worked. If the share doubles, the limit is too close to the live heap. Moving it is a configuration change with no rebuild, which is the best feature of the setting: it can be rolled back by editing an environment variable.

Common mistakes

Setting the limit equal to the container limit. The runtime counts its own memory, not the kernel's view. Stacks of many goroutines, cgo and mapped files push the container over the limit while the Go accounting says there is room. Leave a margin.

Setting it below the steady live heap. The lab shows the result: the program survives but runs several times slower, and nothing in the logs says why unless you read the GC trace. A latency graph that steps up after a deploy that added GOMEMLIMIT is the sign.

Using GOGC=off for a long running service with a leak. With the collector off, nothing happens until the limit, then the collector runs constantly on a heap that is full of live data. A leak that GOGC would have shown as slowly rising memory becomes a cliff.

Forgetting the limit on a short lived tool. A command line tool that processes a large input may prefer GOGC=off with a limit, as in the 517 ms row, because it exits before memory matters. For a tool that runs in a build pipeline, that can cut minutes.

Treating the limit as an alert. It does not log when it is exceeded. Export the metrics and alert on the collector share and on the resident set instead.

A short program to set it in code

Setting the limit from code is useful when the right value depends on the environment, for example when a container limit is read from the cgroup at start:

import "runtime/debug"

func init() {
	// 90 percent of a 1 GiB container.
	debug.SetMemoryLimit(900 << 20)
}

The function returns the previous limit, and passing a negative number only reads the current value. An environment variable is simpler for most deployments, because it can be changed without a build. Code makes sense for a library that must protect itself, or for a program that raises the limit temporarily before a known large allocation and lowers it afterwards.

Checklist

• Set GOMEMLIMIT at about 90 percent of the container limit.
• Keep the default GOGC unless profiling says otherwise.
• Use GOGC=off only for processes that own their memory and have a small live heap.
• A limit below the live heap slows the program, bounded by the GC CPU limiter, instead of hanging it.
• Treat a rising GC share as a signal, and treat the limit as soft.

The feature is described in the Go GC guide and the API in the runtime/debug package.