Microbenchmark Traps in Go That the Compiler Will Spring

Microbenchmarks fail quietly. The number printed is always plausible, always has a unit, and is sometimes the cost of an empty loop. The compiler is better at removing work than most people expect, and the processor is better at predicting a repeated input than a real workload will allow, so the benchmark that looks the most careful can still measure something other than what you asked. This article runs ten variants of two small functions and shows five traps with their numbers: the call that was never made, the constant that flattered the branch predictor, the input that was too kind, the setup billed to the loop, and the result that nothing used.

It continues benchmarking a Go function properly, which gives the method, and it applies to every comparison in this batch, such as maps against slices and sorting. Escape analysis and inlining, which cause some of the effects, are in escape analysis and inlining. The cornerstone is the Go performance guide.

The lab

package main

import (
	"fmt"
	"math/bits"
	"sort"
	"testing"
)

func square(x int) int { return x*x + 1 }

func popcount(x uint64) int {
	n := 0
	for x != 0 {
		x &= x - 1
		n++
	}
	return n
}

func hash(s string) uint32 {
	h := uint32(2166136261)
	for i := 0; i < len(s); i++ {
		h = (h ^ uint32(s[i])) * 16777619
	}
	return h
}

var sinkI int
var sinkU uint32

func bench(name string, f func(b *testing.B)) {
	const runs = 5
	ns := make([]float64, 0, runs)
	for i := 0; i < runs; i++ {
		r := testing.Benchmark(f)
		ns = append(ns, float64(r.T.Nanoseconds())/float64(r.N))
	}
	sort.Float64s(ns)
	fmt.Printf("%-58s %9.3f ns/op\n", name, ns[runs/2])
}

func main() {
	inputs := make([]uint64, 1024)
	x := uint64(88172645463325252)
	for i := range inputs {
		x ^= x << 13
		x ^= x >> 7
		x ^= x << 17
		inputs[i] = x
	}
	fmt.Println("x*x+1, a function the compiler inlines")
	bench("a. result discarded, constant argument (b.N loop)", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			square(7)
		}
	})
	bench("b. result in a global, constant argument", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			sinkI = square(7)
		}
	})
	bench("c. result in a global, argument from a slice", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			sinkI = square(int(inputs[i&1023]))
		}
	})
	fmt.Println()
	fmt.Println("popcount of a 64 bit value, five ways to measure the same function")
	bench("1. result discarded, constant argument (b.N loop)", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			popcount(0xF0F0F0F0F0F0F0F0)
		}
	})
	bench("2. result stored in a global, constant argument", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			sinkI = popcount(0xF0F0F0F0F0F0F0F0)
		}
	})
	bench("3. b.Loop, constant argument", func(b *testing.B) {
		for b.Loop() {
			sinkI = popcount(0xF0F0F0F0F0F0F0F0)
		}
	})
	bench("4. b.Loop, argument cycled through 1024 random values", func(b *testing.B) {
		i := 0
		for b.Loop() {
			sinkI = popcount(inputs[i&1023])
			i++
		}
	})
	bench("5. the same, with math/bits.OnesCount64 (a CPU instruction)", func(b *testing.B) {
		i := 0
		for b.Loop() {
			sinkI = bits.OnesCount64(inputs[i&1023])
			i++
		}
	})
	fmt.Println()
	fmt.Println("FNV hash of a string, the effect of the input")
	short := "key"
	long := "a considerably longer key used for a lookup in the table"
	bench("6. b.Loop, constant 3 byte string", func(b *testing.B) {
		for b.Loop() {
			sinkU = hash("key")
		}
	})
	bench("7. b.Loop, variable 3 byte string", func(b *testing.B) {
		for b.Loop() {
			sinkU = hash(short)
		}
	})
	bench("8. b.Loop, variable 55 byte string", func(b *testing.B) {
		for b.Loop() {
			sinkU = hash(long)
		}
	})
	fmt.Println()
	fmt.Println("setup inside or outside the timed region")
	bench("9. build a 1000 element slice in the loop, then sum", func(b *testing.B) {
		for b.Loop() {
			s := make([]int, 1000)
			for i := range s {
				s[i] = i
			}
			t := 0
			for _, v := range s {
				t += v
			}
			sinkI = t
		}
	})
	s := make([]int, 1000)
	for i := range s {
		s[i] = i
	}
	bench("10. build once outside, sum in the loop", func(b *testing.B) {
		for b.Loop() {
			t := 0
			for _, v := range s {
				t += v
			}
			sinkI = t
		}
	})
}

x*x+1, a function the compiler inlines
a. result discarded, constant argument (b.N loop)              0.318 ns/op
b. result in a global, constant argument                       0.315 ns/op
c. result in a global, argument from a slice                   0.675 ns/op

popcount of a 64 bit value, five ways to measure the same function
1. result discarded, constant argument (b.N loop)             11.361 ns/op
2. result stored in a global, constant argument               11.310 ns/op
3. b.Loop, constant argument                                  12.548 ns/op
4. b.Loop, argument cycled through 1024 random values         26.587 ns/op
5. the same, with math/bits.OnesCount64 (a CPU instruction)     2.060 ns/op

FNV hash of a string, the effect of the input
6. b.Loop, constant 3 byte string                              3.466 ns/op
7. b.Loop, variable 3 byte string                              3.464 ns/op
8. b.Loop, variable 55 byte string                            38.028 ns/op

setup inside or outside the timed region
9. build a 1000 element slice in the loop, then sum          895.619 ns/op
10. build once outside, sum in the loop                      323.955 ns/op

Trap 1: the call that was never made

The first block benchmarks square(x) = x*x + 1, a function small enough to be inlined:

A multiplication and an add cost about a third of a nanosecond on this machine only if the work is really done and perfectly overlapped, and the first two rows are what an empty loop costs: one increment and one branch per iteration, about one cycle. The compiler inlined square(7), saw that the argument was a constant and replaced the whole expression by 50. In variant a it then deleted the unused result. In variant b it kept a store to the global but stored a constant. In both the benchmark loop timed nothing but itself. Variant c, with an argument it could not know in advance, did the arithmetic and a memory load, and took twice as long.

The lesson is not that 0.3 ns is a bad result. It is that a result below one nanosecond for a function that does anything should make you suspicious, and that the way to confirm is to look at the generated code (go test -bench X -gcflags=-S, or go tool objdump on the test binary) or to change the input and see whether the time moves.

Trap 2: the constant that the branch predictor learns

Popcount counts the set bits in a word with a loop that runs once per set bit. The inputs decide how many iterations the loop runs, which the processor tries to predict:

Here the compiler did not remove the call, since a loop with a data dependent count is not folded, and all three constant variants (1 to 3) agree at 11 to 12.5 ns. But the constant argument means the loop runs exactly 32 times, every time, and the processor learns the pattern after a few iterations and never mispredicts the exit. Feed it values with different bit counts and the exit branch is unpredictable: the same function took 26.6 ns, 2.1 times longer. The benchmark with the constant was accurate about the code and wrong about the workload.

Variant 5 shows what the standard library does for you: bits.OnesCount64 compiles to a single instruction and took 2.06 ns including the load and loop, thirteen times less than the hand written loop on real inputs. A benchmark that finds you have been writing a slower copy of the library is worth running.

Branch behaviour is the most common reason a microbenchmark disagrees with production. Sorted input, repeated keys and a handful of distinct values all make branches predictable, and a service that sees varied data does not get that help. Benchmark with inputs drawn from a realistic distribution, cycle through many of them, and shuffle if order could matter.

Trap 3: the input that is too small or too kind

The constant string and the variable string cost the same here, because the loop over three bytes is not removed, so constant folding was not the trap this time. The trap is the size: hashing cost grows with the key, 11 times for 18 times the bytes. A benchmark that hashes "key" reports 3.5 ns and tells you nothing about the URLs and identifiers the service really sees. Measure at the sizes you have. Take them from production logs or a sample, and report more than one.

Trap 4: setup billed to the thing being measured

The first measures allocation, zeroing, filling and summing together. The second measures the sum, and the 570 ns difference is the cost of building. Neither number is wrong. They answer different questions, and the mistake is to report the first as "the cost of summing". If the question is how long the whole operation takes in a service that builds fresh data each time, then variant 9 is right. Decide what is in the question before deciding what is in the loop, and with b.Loop put only that inside.

Trap 5: the result that nothing used

The b.N loop in variant a discarded the result, and the compiler removed the call. The older advice was to assign the result to a package variable, as variants b and c do, and the lab shows both its value (variant c was not eliminated) and its limit (variant b stored a constant and was still empty). Since Go 1.24 for b.Loop() keeps the call and its result alive, so the discarding trap is closed for code that uses it. Variants 3 to 10 use it, and you can see it in the numbers: variant 3, the constant popcount, kept its work.

It does not close the folding trap. b.Loop prevents the compiler from deleting a call. It does not prevent the compiler from computing a constant expression at build time if the arguments are constants and the function is simple enough to inline and evaluate. A benchmark with square(7) inside b.Loop may still be cheap for that reason. Vary the input, and the folding goes away.

A checklist before believing a number

1. Is it faster than the hardware could plausibly be? A result under about 1 ns for real work means the work was removed.
2. Does it change when the input changes? Run with constant, small, large and random inputs. If the number does not move with the input, either the function is flat or the input is being ignored.
3. Did you vary the data enough to defeat the branch predictor and the cache? A thousand values in rotation did here.
4. Is anything in the loop that is not in your question? Allocation, formatting, a time.Now() call, logging.
5. Is the result used? With b.Loop yes. With b.N, assign it to a global.
6. Do the allocation counts make sense? -benchmem showing zero allocations for code that builds a string means it was removed.
7. Does the benchmark run long enough, several times? Use -count=6 and benchstat.
8. Does a profile of the benchmark show the code you meant? -cpuprofile and go tool pprof -top should show your function.

Checklist

• Use for b.Loop() and realistic, varied inputs.
• Distrust results near the cost of an empty loop (about 0.3 ns here).
• A constant input flatters branches and caches; cycle through many values.
• Benchmark at the input sizes your service sees.
• Keep setup out of the loop unless it belongs to the question.
• Compare with the standard library version when one exists.

The testing.B.Loop behaviour is documented in the testing package, and the compiler's inlining decisions can be listed with -gcflags=-m as described in the compiler diagnostics.