Profile and Speed Up Go Programs With pprof and Benchmarks

Performance work in Go has a reputation for being either trivial (add a cache, add a goroutine) or arcane (cache lines, escape analysis, assembly). In practice it is neither. Almost all of it is a loop with four steps: measure the program, find where the time or memory goes, change that one place, and check that the change helped and broke nothing. The tools for each step ship with the language. What separates programmers who get results from those who do not is mostly discipline: they measure before they change, they change one thing at a time, and they believe the numbers when the numbers disagree with their intuition.

This page is the map to the rest of the performance cluster. It is long on purpose and built so you can leave after any section. The first section is the short version. The second walks one small program through the whole loop with real profiles and real numbers, and the sections after that group the other articles by the question they answer, with the finding from each so you can decide whether to read it. At the end are a triage table, three worked decisions and a checklist.

The short version

If you read nothing else, these rules cover most of what goes wrong.

1. Measure before you change anything. A benchmark with allocation counts, run several times and compared with benchstat, is the only evidence that counts. See benchmarking a function properly.
2. Profile instead of guessing. A CPU profile says where the time goes and an allocation profile says where the garbage comes from, and both are usually surprising. See CPU profiling with pprof and allocation profiles.
3. Fix the biggest site first, and stop when the profile is flat. One or two functions account for most of the cost in almost every program. See reducing allocations.
4. Prove behaviour is unchanged. A hash of the output or a test that runs old and new on the same input. A fast wrong answer is worse than a slow right one.
5. Distrust microbenchmarks. The compiler can delete your code, a constant input flatters the branch predictor, and setup can leak into the timer. See microbenchmark traps.
6. Allocations matter in aggregate, not individually. Watch bytes and allocation counts per operation and the collector's share of CPU, not a single allocation. See GOGC tuning and GC pauses.
7. The pause is rarely the problem. Go's pauses were tens of microseconds at every heap size we measured. The cost is mark work and assists, so look at the GC CPU share.
8. Memory that stays high is not always a leak. Read the live heap after collection, not RSS. See RSS against the heap and memory leaks that are not goroutines.
9. Batch your system calls. A call costs hundreds of nanoseconds whatever it carries. See the cost of syscalls and buffered IO.
10. Do not tune what is not slow. If a function is under a few percent of the profile, leave it readable.

One program, end to end

The best way to see the loop is to run it. The program counts the most common words in 2 MB of generated text and returns the top 20. The first draft is the one most people would write: lower case the text, find words with a regexp, count them in a map, sort. A test checks that every version returns the same answer, because every step below must be a rewrite and not a change of behaviour.

package words

import (
	"regexp"
	"sort"
	"strings"
)

var wordRE = regexp.MustCompile(`[A-Za-z']+`)

// Slow is the first draft: lower case the text, find words with a regexp, count in a map.
func Slow(text string, top int) []string {
	counts := map[string]int{}
	for _, w := range wordRE.FindAllString(strings.ToLower(text), -1) {
		counts[w]++
	}
	return rank(counts, top)
}

// Fast walks the bytes once, lower cases into a reused buffer and counts without a slice of matches.
func Fast(text string, top int) []string {
	counts := make(map[string]int, 4096)
	buf := make([]byte, 0, 32)
	flush := func() {
		if len(buf) > 0 {
			counts[string(buf)]++
			buf = buf[:0]
		}
	}
	for i := 0; i < len(text); i++ {
		c := text[i]
		switch {
		case c >= 'A' && c <= 'Z':
			buf = append(buf, c+'a'-'A')
		case c >= 'a' && c <= 'z', c == '\'':
			buf = append(buf, c)
		default:
			flush()
		}
	}
	flush()
	return rank(counts, top)
}

// Faster keeps a pointer to each count, so a word already seen is found with a lookup the
// compiler can do without building a string, and only a new word allocates its key.
func Faster(text string, top int) []string {
	counts := make(map[string]*int, 4096)
	buf := make([]byte, 0, 32)
	flush := func() {
		if len(buf) == 0 {
			return
		}
		if p := counts[string(buf)]; p != nil {
			*p++
		} else {
			n := 1
			counts[string(buf)] = &n
		}
		buf = buf[:0]
	}
	for i := 0; i < len(text); i++ {
		c := text[i]
		switch {
		case c >= 'A' && c <= 'Z':
			buf = append(buf, c+'a'-'A')
		case c >= 'a' && c <= 'z', c == '\'':
			buf = append(buf, c)
		default:
			flush()
		}
	}
	flush()
	flat := make(map[string]int, len(counts))
	for w, p := range counts {
		flat[w] = *p
	}
	return rank(flat, top)
}

func rank(counts map[string]int, top int) []string {
	type kv struct {
		w string
		n int
	}
	list := make([]kv, 0, len(counts))
	for w, n := range counts {
		list = append(list, kv{w, n})
	}
	sort.Slice(list, func(i, j int) bool {
		if list[i].n != list[j].n {
			return list[i].n > list[j].n
		}
		return list[i].w < list[j].w
	})
	if top > len(list) {
		top = len(list)
	}
	out := make([]string, top)
	for i := range out {
		out[i] = list[i].w
	}
	return out
}

package words

import (
	"math/rand/v2"
	"reflect"
	"strconv"
	"strings"
	"testing"
)

func corpus() string {
	r := rand.New(rand.NewPCG(9, 9))
	vocab := make([]string, 3000)
	for i := range vocab {
		vocab[i] = "Word" + strings.Repeat("x", i%7) + strings.Map(func(c rune) rune { return 'a' + (c-'0')%26 }, strconv.Itoa(i))
	}
	var sb strings.Builder
	for sb.Len() < 2<<20 {
		sb.WriteString(vocab[r.IntN(len(vocab))])
		sb.WriteString([]string{" ", ", ", ". ", "\n", " - "}[r.IntN(5)])
	}
	return sb.String()
}

func TestSame(t *testing.T) {
	text := corpus()
	want := Slow(text, 20)
	for name, f := range map[string]func(string, int) []string{"Fast": Fast, "Faster": Faster} {
		if got := f(text, 20); !reflect.DeepEqual(want, got) {
			t.Fatalf("%s differs:\n%v\n%v", name, want, got)
		}
	}
}

var sink []string

func BenchmarkSlow(b *testing.B) {
	text := corpus()
	b.ReportAllocs()
	b.SetBytes(int64(len(text)))
	for b.Loop() {
		sink = Slow(text, 20)
	}
}

func BenchmarkFast(b *testing.B) {
	text := corpus()
	b.ReportAllocs()
	b.SetBytes(int64(len(text)))
	for b.Loop() {
		sink = Fast(text, 20)
	}
}

func BenchmarkFaster(b *testing.B) {
	text := corpus()
	b.ReportAllocs()
	b.SetBytes(int64(len(text)))
	for b.Loop() {
		sink = Faster(text, 20)
	}
}

The script does what you would do at a terminal. It runs the benchmark of the slow version and writes a CPU and a memory profile, reads the top of each with go tool pprof, then benchmarks all three versions six times each and compares them with benchstat.

set -e
GOBIN=$PWD/bin go install golang.org/x/perf/cmd/[email protected]
go test -run TestSame -count=1 .
echo "--- 1. measure: benchmark the slow version and write profiles"
go test -run '^$' -bench Slow -benchtime=2s -cpuprofile cpu.out -memprofile mem.out -o words.test .
echo "--- 2. CPU profile, top of the list"
go tool pprof -top -nodecount=9 words.test cpu.out 2>/dev/null | grep -vE '^(Time|Duration):'
echo "--- 3. allocation profile, bytes allocated"
go tool pprof -sample_index=alloc_space -top -nodecount=5 words.test mem.out 2>/dev/null | grep -vE '^(Time|Duration):'
echo "--- 4. compare the three versions, 6 runs each"
go test -run '^$' -bench 'Slow$' -benchtime=300ms -count=6 . > old.txt
go test -run '^$' -bench 'Fast$' -benchtime=300ms -count=6 . | sed 's/BenchmarkFast/BenchmarkSlow/' > fast.txt
go test -run '^$' -bench 'Faster$' -benchtime=300ms -count=6 . | sed 's/BenchmarkFaster/BenchmarkSlow/' > faster.txt
./bin/benchstat old.txt fast.txt faster.txt
rm -rf bin old.txt fast.txt faster.txt cpu.out mem.out words.test

ok  	example.test/lab	0.359s
--- 1. measure: benchmark the slow version and write profiles
goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
BenchmarkSlow-10    	      25	  80385245 ns/op	  26.09 MB/s	19360584 B/op	  168916 allocs/op
PASS
ok  	example.test/lab	2.382s
--- 2. CPU profile, top of the list
File: words.test
Type: cpu
Showing nodes accounting for 1460ms, 79.35% of 1840ms total
Showing top 9 nodes out of 98
      flat  flat%   sum%        cum   cum%
     390ms 21.20% 21.20%      390ms 21.20%  runtime.madvise
     290ms 15.76% 36.96%      350ms 19.02%  regexp.(*machine).add
     200ms 10.87% 47.83%      660ms 35.87%  regexp.(*machine).step
     150ms  8.15% 55.98%      150ms  8.15%  regexp/syntax.(*Inst).MatchRunePos
     130ms  7.07% 63.04%      130ms  7.07%  runtime.memmove
     100ms  5.43% 68.48%      910ms 49.46%  regexp.(*machine).match
      90ms  4.89% 73.37%       90ms  4.89%  regexp.(*inputString).step
      60ms  3.26% 76.63%      130ms  7.07%  regexp.(*Regexp).FindAllString.func1
      50ms  2.72% 79.35%       50ms  2.72%  internal/runtime/atomic.(*UnsafePointer).Load (inline)
--- 3. allocation profile, bytes allocated
File: words.test
Type: alloc_space
Showing nodes accounting for 468.25MB, 98.60% of 474.90MB total
Dropped 35 nodes (cum <= 2.37MB)
Showing top 5 nodes out of 13
      flat  flat%   sum%        cum   cum%
  334.54MB 70.44% 70.44%   334.54MB 70.44%  regexp.(*Regexp).FindAllString.func1
   66.50MB 14.00% 84.45%    67.50MB 14.21%  regexp.(*Regexp).doExecute
   51.12MB 10.76% 95.21%    51.12MB 10.76%  strings.(*Builder).grow
   11.99MB  2.52% 97.74%    11.99MB  2.52%  strings.(*Builder).WriteString
    4.10MB  0.86% 98.60%   457.80MB 96.40%  example.test/lab.Slow
--- 4. compare the three versions, 6 runs each
goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
        │   old.txt    │              fast.txt              │             faster.txt             │
        │    sec/op    │   sec/op     vs base               │   sec/op     vs base               │
Slow-10   77.417m ± 4%   8.942m ± 6%  -88.45% (p=0.002 n=6)   7.074m ± 4%  -90.86% (p=0.002 n=6)

        │   old.txt    │               fast.txt                │              faster.txt               │
        │     B/s      │      B/s       vs base                │      B/s       vs base                │
Slow-10   25.84Mi ± 4%   223.67Mi ± 5%  +765.74% (p=0.002 n=6)   282.73Mi ± 4%  +994.35% (p=0.002 n=6)

        │    old.txt     │               fast.txt               │             faster.txt              │
        │      B/op      │     B/op       vs base               │     B/op      vs base               │
Slow-10   18909.2Ki ± 0%   2636.9Ki ± 0%  -86.05% (p=0.002 n=6)   458.1Ki ± 0%  -97.58% (p=0.002 n=6)

        │    old.txt    │              fast.txt               │             faster.txt             │
        │   allocs/op   │   allocs/op    vs base              │  allocs/op   vs base               │
Slow-10   168.923k ± 0%   168.849k ± 0%  -0.04% (p=0.002 n=6)   6.031k ± 0%  -96.43% (p=0.002 n=6)

Step 1: measure

The benchmark of the first draft reported 80.4 ms per run, 19,360,584 bytes and 168,916 allocations for 2 MB of text, 26 MB/s. That is the baseline, and every claim below is relative to it. Two lines of the output deserve attention before any profile is read: the throughput is small for a text scan (a quarter of what a plain loop over bytes reaches), and the allocation count, 168,916, is close to the number of words in the text. One allocation per word is the signature of a slice of matches.

Step 2: the CPU profile

The top of the CPU profile reads:

Three things to read. Flat and cumulative differ. Flat time is spent in the function itself and cumulative includes what it calls. regexp.(*machine).match has little flat time and 49.5 percent cumulative, so it is the root of half of the profile: the regexp engine is where the time goes, not any one line in our code. The first row is a warning, not a function. runtime.madvise is the runtime handing memory back to the operating system, and 21 percent of the profile in it means the program is allocating and freeing memory at a rate the runtime has to manage, which is also what the allocation count suggested. A profile of a healthy compute bound program does not look like that. The shape is a regexp problem. The machine functions are the regexp engine's automaton, the cost the article on why regular expressions are slow describes: a safe engine that steps through states for every byte.

For how to read the full tree, including list, web and the flame graph, see reading a pprof flame graph.

Step 3: the allocation profile

Sorted by bytes allocated, the profile says:

One function, FindAllString, accounts for 70 percent of everything allocated: it builds a slice of every match, each of them a string. That is the site. The Builder rows are strings.ToLower creating a lower cased copy of 2 MB. No amount of tuning the garbage collector would have solved this, because the program makes the garbage on purpose.

Step 4: change the one thing

Fast removes the regexp, the ToLower copy and the slice of matches. It walks the text byte by byte, appends lower cased letters to a reused buffer, and when it meets a non letter it counts the word and resets the buffer. The result, from six runs each:

The p=0.002 is the smallest value six samples against six can give. The ± columns show 4 to 7 percent noise, so differences of 1 or 2 percent in this table would mean nothing, and the ones above are far outside it.

Step 5: read what did not move

The surprise is the allocation column for Fast. Time fell by 88 percent and bytes by 86, and allocations did not move at all: 168,849 against 168,923. The cause is one line, counts[string(buf)]++. Go avoids copying for a map lookup written as m[string(b)], but the increment form needs an assignment, which stores the key, so it builds a new string for every word, repeats included. The byte total fell because those strings are small and the 2 MB lower cased copy and the slice of matches are gone, and the count stayed because the keys remained.

This is the value of reading the full benchmark output instead of the headline. Faster stores a pointer to the count in the map. A word already present is found with the allocation free lookup, the count is incremented through the pointer, and only a first occurrence allocates its key. The text has 3,000 distinct words, and allocations fell to 6,031, 96 percent below the draft. The time improved from 8.9 ms to 7.1 ms, a smaller gain than the allocation count suggests, which fits the earlier lesson: once the big cost is gone, the rest of the allocations are a modest share of a program that now spends its time scanning bytes. The next step in a real project would be a new profile, and probably the decision to stop.

Three habits are visible in this small loop. The test ran every version against the same input before any number was believed. The profile named the site (one function, 70 percent) before any code changed. And each change was measured separately, so the table shows what the regexp removal bought (88 percent) and what the allocation clean up bought (a further 2.4 points of the baseline). Without that, the whole improvement would have been credited to a single rewrite.

Choosing the tool

The tools fit questions, and using the wrong one wastes an afternoon.

Profiling

CPU profiling is the starting point: attach the profiler for a few seconds, then read the top functions by cumulative time. Reading a flame graph covers the visual form of the same data and the four mistakes people make with it, and in particular the habit of staring at the widest bar instead of the widest bar that you control. Heap profiles answer a different question, what is holding memory right now, and allocation profiles answer where garbage comes from, which is the question behind most GC trouble.

The practical details of profiling a service are in the pprof handlers, which found that a 2 second profile succeeds on a server whose write timeout is 1 second in Go 1.25 because the handler extends its own deadline, and that the handlers must never be exposed on the public listener. Continuous profiling is the production version: a low overhead sample, always on, so that the afternoon the incident happens you already have the profile. The execution tracer is the tool for latency and scheduling: it shows goroutines waiting, not only running, which no sampling profile can.

Letting the compiler help

Escape analysis explains why a value is allocated on the heap. In the lab, a variable sized make allocated even though the compiler's output said the slice did not escape, a case worth knowing before you trust -m output as a promise. Inlining and bounds check elimination both measured as no gain on the benchmark we ran on this machine, which is the honest summary: they are compiler behaviours to read about when a profile points at them, and not a place to hunt for speed blindly.

Profile guided optimization is the compiler using your profile. Building with a profile from the real binary gave a 3 percent improvement (2,816 ms to 2,732 ms) at 2.6 percent larger binary size, and a profile taken from a test binary had no effect at all because its function names did not match. It is free once the pipeline exists, and modest.

Where allocations come from and how to cut them

Allocation work is the largest part of the cluster because it is where most Go programs have easy wins. Reducing allocations is the method in four measured steps: presizing and a Builder saved 5 percent, replacing strings.Split with strings.Cut removed 2,007 of 2,171 allocations and made the job 2.1 times faster, and reusing scratch space removed the rest, with a hash check at each step.

The building blocks have their own articles. Preallocating slices and maps shows the cost of growth. String concatenation and the strings Builder compare plus, Join and Builder, and explain why a pool of Builders pools nothing. String and byte conversions catalogues where string(b) copies and where the compiler avoids it, which is the same effect that decided the lab above.

For memory layout, struct padding and alignment took a 40 byte struct to 24 by reordering fields, 160 MB to 96 MB across four million values, with a scan only 3 percent faster, because the saving is memory and the speed depends on the access pattern. False sharing is the case where padding is added on purpose: ten goroutines incrementing their own counters took 5,876 ms side by side and 181 ms on separate cache lines, 32 times faster, with no change to the logic.

sync.Pool shows the best case and the cost: a 64 KB buffer took 4,787 ns from make and 8.4 ns from a pool, but the benchmark flatters it because the collector never cleared the pool, and oversized or unreset buffers leak data and memory. Arenas and memory pools compares the experimental arena package, a plain slab and new on a million node tree: 1.41 s, 1.01 s with an arena and 0.60 s with the slab, and a use after free that read stale data silently. The conclusion there is typical of the cluster: the safe, boring technique beat the exotic one.

The garbage collector and memory

The collector has four articles, in the order you will meet the problems.

GOGC tuning measures the knob: raising it from 100 to 400 took a 600,000 allocation job from 2,064 ms to 500 ms and 18 collections to 3, at the price of a heap of 871 MB against 271 MB. GOMEMLIMIT adds the memory budget, and shows that a limit below the live heap slows the program, 3,214 ms against 517 ms, and does not hang it, because a CPU limiter caps the collector. Garbage collector pauses puts the common worry to rest: pauses were tens of microseconds at 50, 200 and 800 MB, and the real cost of a pointer rich heap is mark work, ten times the wall time of the same heap held as bytes.

Then there are the questions that arrive as incidents. Memory looks high separates what your code holds, what the runtime holds and what the kernel counts: after 512 MB was dropped and collected, the heap said 0 and RSS said 517 MB, and on Linux debug.FreeOSMemory returned it all while on macOS RSS did not move. Memory leaks that are not goroutines builds four (an unbounded map, a substring that pins its parent string, uncancelled contexts, closures that capture buffers) and measures each, 105 MB to 477 MB for 100,000 iterations, with a one line fix for each. Slices that keep huge arrays alive is a closer look at the commonest, where 200 slices of 32 bytes held 205 MB, and maps that never shrink shows that deleting 999,990 entries from a million entry map freed nothing.

The shared lesson is to watch the right number: live heap after collection for leaks, the GC share of CPU for collector cost, and RSS for what the container limit sees.

Benchmarking and the comparisons

Benchmarking a function properly and microbenchmark traps are the methodology. The traps are specific: an inlined function with a constant argument timed 0.3 ns, the cost of the loop, and a popcount took 12.5 ns on a constant and 26.6 ns on varied values because the branch predictor learned the constant.

The remaining articles apply the method to common choices, each with a table and a verdict:

• Maps against slices: for ints they tie to about 16 entries; for strings the map wins from about 8; building a 64 entry map costs 11 times a slice.
• JSON libraries: goccy decoded 21 KB 8.9 times faster than encoding/json, with 6 allocations against 536, and json-iterator was slower at encoding.
• Routers: from 37 ns (gin) to 307 ns (chi) for a parameter route with 50 routes, all negligible next to a handler.
• Hash functions: xxhash at 14.5 GB/s and FNV at 0.8 GB/s on 64 KB, and SHA-256 faster than MD5 on hardware with SHA instructions.
• Sorting: slices.Sort matched sort.Ints, sort.Slice was 66 percent slower, and a stable sort cost 3.5 to 10 times more.
• Regexp against manual parsing and why regexp is slow: a log line took 1,246 ns with a regexp and 37 ns by hand, and a catastrophic pattern took Python 9 seconds where Go took microseconds, with a 66 times cliff for case insensitive matching.
• Fast string search: 63 GB/s for one byte and 4 GB/s for a literal.
• Faster JSON and Protobuf against JSON: narrow structs, streaming and hand written encoders for the first, a 44 percent smaller and up to 5.5 times faster decoding format for the second.

Input and output

I/O is where most of the time goes in real services, and its costs are mostly about call counts. Reading large files measures buffer sizes (512 bytes took 564 ms for a 512 MB file and 32 KB took 51 ms), the allocation per line of ReadString, and the danger of ReadFile. Parsing big CSV files uses ReuseRecord to cut 115 MB of allocation to 35 MB, and a no copy scan to 71 ms. Buffered IO explains the 49 times difference between a write per line and a buffered writer, and where wrapping does nothing. Memory mapped files gives random reads at 7 ns instead of 645 ns and lists the costs. And syscalls is the reason underneath: a call cost 241 ns on Linux and 575 ns on macOS for a write of any size, so 65,536 one byte writes took 38 ms and one 64 KB write took 573 ns.

Database costs belong to the databases cluster, not this one; start with the SQL pool settings, since a pool of 2 against 20 connections changed a job from 120 ms to 1,040 ms there.

A triage table

Three decisions, worked

"Should I replace encoding/json?" First a profile of the service. If JSON is under 10 percent of CPU, no. If it is 30 percent, try the cheap changes: a narrow struct for the fields you read (1,837 ns against 3,422 ns for a map decode), a streaming decoder for large arrays (155 MB of peak heap against 296 MB), and omitzero and smaller payloads. If a profile still shows JSON dominant after those, a faster decoder is worth testing behind an interface with golden tests, because an 8.9 times gain is available for decoding, and then read what you are giving up: behaviour at the edges, another dependency, and an experimental encoding/json/v2 coming with the language.

"Why is memory 600 MB when the heap profile says 80?" Not a leak yet. Read HeapAlloc (what the program holds), HeapIdle and HeapReleased (what the runtime keeps), and RSS (what the kernel counts). A gap that closes slowly after traffic falls is the scavenger. A live heap that climbs between collections is a leak, and then two heap profiles an hour apart and go tool pprof -base find the type, though the allocation site is only a clue to the holder. Set GOMEMLIMIT at 90 percent of the container limit so the runtime works harder before the kernel intervenes.

"The service is slow under load, but no function stands out." Take an execution trace and a CPU profile at load. If the CPU profile is flat and the trace shows goroutines waiting, the problem is waiting: a lock, a pool of connections that is too small, a channel with no consumer, a slow downstream. If the GC share is above 20 percent, look at allocation rate and live heap. If both are quiet and the time is in system calls, count them. A flat profile is information: the cost is not in your code.

Reading a benchmark diff honestly

The benchstat table in the lab is short, and every column in it has a way of being misread. The centre value is the median of the samples, so one slow outlier does not move it, and the ± figure is the spread of the samples as a percentage of it: 4 to 7 percent here, which is what a quiet laptop gives. A change smaller than the spread is not a result, whatever the median says. The p value is the probability of seeing a difference this large if the two groups of runs came from the same distribution, and six samples against six cannot give less than 0.002, so a table with p=0.002 everywhere says only that the effect is large enough to be consistent, not that it is exactly the size printed. When benchstat prints a tilde instead of a percentage, it is telling you the two versions cannot be told apart, and the right conclusion is that the change did nothing, not that you need more runs until it does.

Three more habits keep the table honest. Compare like with like: both versions on the same machine, in the same session, with the same input and the same Go version. Report all the columns, as the lab does, because the time, the bytes and the allocation count can move independently, and the Fast row, where time fell 88 percent and allocations did not move, is the case that shows why. And keep the raw output. A claim that a change is "2.1 times faster" is much more useful in a pull request with the two files attached, since a reviewer can rerun it on their machine.

Habits of people who get results

The people who make programs faster for a living do not know more tricks than everyone else. They share a handful of habits, and the cluster's articles are mostly worked examples of these.

They start from a question about the user, not about the code. "The checkout page takes 900 ms at the 99th percentile and the target is 400" is a question a profile can answer. "This function feels slow" is not.

They keep a baseline. Before changing anything they save the benchmark output and the profile, so that after the change they can show the difference and, if it did not help, return to a known state.

They change one thing at a time. The lab ran three versions and reported three rows, so the credit for each step is visible. A commit that replaces the regexp, adds a pool and changes the GOGC setting cannot say which of them helped, and often one of the three made things worse while the other two covered for it.

They look at the full profile before the top line. The first row in the lab's CPU profile was a runtime function, not the regexp, and it was the clue that the program was allocating heavily. Reading only the first entry that is "your code" would have missed it.

They know what each number measures. Flat against cumulative, alloc_space against inuse_space, HeapAlloc against RSS, wall time against CPU time. Half of the confusion in performance discussions is two people using the same word for different things.

They are comfortable with an answer of "leave it". The profile is flat, the function is under two percent, the gain is 3 percent for a 40 line change that nobody will understand in a year: leave it, write down why, and move on. The cluster's measurements of inlining, bounds checks and pooled buffers are all of this kind.

They protect the gain. A benchmark in the repository, an allocation assertion in a test, a dashboard panel for the GC share, a comment at the optimised code. Performance regressions are usually accidents, and accidents are prevented by checks that run without anyone remembering them.

A first week with a slow service

If you inherit a slow service and are not sure where to start, a plan that has worked: on the first day, expose the pprof handlers on a private port, capture a 30 second CPU profile and a heap profile at normal load, and add the live heap, GC share, allocation rate and RSS to the dashboard. On the second day, read the CPU profile top to bottom, write down the three functions with the highest cumulative time that you own, and take an allocation profile. On the third day, write a benchmark for the most expensive one with real inputs and record the baseline. On the fourth and fifth days, make one change at a time, each with its before and after table, and stop when the profile is flat or the target is met. At the end of the week you have a smaller set of problems, evidence for each decision, and the instruments to see the next regression the day it arrives.

What changes with scale

At small scale performance work pays when it removes a visible slowness, and not otherwise. A tool that runs for 50 ms has no use for a faster sort. At the scale of a service handling thousands of requests a second, the same 50 µs saved is real CPU, fewer machines, and a lower tail, and the cheap, broad techniques (fewer allocations, batched I/O, a sane GOGC and GOMEMLIMIT) pay for themselves. At the scale of a fleet, continuous profiling is the way to find the 2 percent costs that sum to a server, and PGO and library level choices start to matter. At every scale the order is the same: measure, find, fix, verify.

Where this guide does not apply

This cluster is about CPU and memory in a single Go process. It does not cover distributed systems latency (queues, retries, fan out), where the cost is a network, nor database query tuning, where the fix is an index, nor frontend performance. It also does not tell you what to optimise: that depends on your service level objectives, which are about users, and not about nanoseconds. A faster function in a request that spends 95 percent of its time in a database call changes nothing a user can see.

The measurements come from an Apple M1 Max, a machine with wide cores and large caches and with hardware instructions for hashing and CRC. Ratios are the portable part. Absolute figures (nanoseconds per call, GB per second) are properties of this machine, and a different CPU, kernel or Go release can change them, sometimes by a factor of two. Run the lab on your own hardware before you quote a number.

A production checklist

• Benchmarks exist for the hot paths, run with -count and compared with benchstat, and allocation counts are tracked with testing.AllocsPerRun in a test.
• net/http/pprof is available on a private port, and continuous profiling runs in production.
• Dashboards show the live heap after collection, GC CPU share, allocation rate and RSS separately.
• GOMEMLIMIT is set to about 90 percent of the container limit, and GOGC is changed only after a profile.
• Output equivalence is tested before and after every performance change.
• Hot paths avoid per item allocation, system calls per small piece and reflection where a profile says it matters, and nowhere else.
• Comments on optimised code say what it avoids and point to the benchmark.
• Every claim in a pull request has a benchstat table behind it.

Where to go next

If you have a slow program in front of you, start with CPU profiling and benchmarking a function. If you have a memory problem, start with RSS against the heap and heap profiles. If you are choosing between libraries, read the benchmark articles above, and the methodology article first, so you can check the numbers against your own workload. If you are curious how far the language goes, false sharing, struct layout and arenas are the deep end, and the honest conclusion of each is that the simple version is usually enough.

The official references are the Go diagnostics guide, the garbage collector guide and the pprof documentation.