Benchmark a Go Function Properly From Setup to Result

A benchmark that prints one number is a guess with a decimal point. The first run of the same function in the lab below showed 534.9 ns and the next five showed 502 to 520, which is noise that a single run would have turned into a conclusion. A trustworthy result needs a benchmark that cannot be optimised away, several samples, a comparison that says how likely the difference is to be chance, and the allocation counts that explain it. This article builds that for one function, a slug generator written two ways, and shows each step: the benchmark function, b.Loop, sub-benchmarks for input sizes, -benchmem, -count, and benchstat to read the result.

It is the base for microbenchmark traps, and every comparison in this batch (maps against slices, JSON libraries, sorting) uses the method. For finding out what to benchmark in the first place, start from a CPU profile. The cornerstone is the Go performance guide.

The function under test

A slug turns "Benchmark A Go Function, Properly!" into benchmark-a-go-function-properly. The first version appends one character at a time to a string. It is correct and quadratic: every += allocates a new string and copies the old one. The second writes bytes into a strings.Builder sized for the input. A test checks that the two agree on ASCII input, which is the rule for any benchmark comparison: measure only things that produce the same answer.

package slug

import "strings"

// SlugConcat builds the result by appending to a string, one allocation per character kept.
func SlugConcat(s string) string {
	out := ""
	dash := false
	for _, r := range strings.ToLower(s) {
		switch {
		case r >= 'a' && r <= 'z', r >= '0' && r <= '9':
			out += string(r)
			dash = false
		default:
			if !dash && out != "" {
				out += "-"
				dash = true
			}
		}
	}
	return strings.TrimSuffix(out, "-")
}

// SlugBuilder does the same job with one pre-sized buffer.
func SlugBuilder(s string) string {
	var b strings.Builder
	b.Grow(len(s))
	dash := false
	for i := 0; i < len(s); i++ {
		c := s[i]
		switch {
		case c >= 'A' && c <= 'Z':
			c += 'a' - 'A'
			fallthrough
		case c >= 'a' && c <= 'z', c >= '0' && c <= '9':
			b.WriteByte(c)
			dash = false
		default:
			if !dash && b.Len() > 0 {
				b.WriteByte('-')
				dash = true
			}
		}
	}
	return strings.TrimSuffix(b.String(), "-")
}

The benchmark

package slug

import (
	"fmt"
	"strings"
	"testing"
)

func TestSame(t *testing.T) {
	for _, in := range []string{"", "Hello, World!", "  Go 1.25 -- Benchmarks  ", "ÅÄÖ x"} {
		if in == "ÅÄÖ x" {
			continue // the byte version is ASCII only by design
		}
		if a, b := SlugConcat(in), SlugBuilder(in); a != b {
			t.Fatalf("%q: %q != %q", in, a, b)
		}
	}
}

func input(n int) string {
	return strings.Repeat("Benchmark A Go Function, Properly! ", n/35+1)[:n]
}

var sink string

func BenchmarkSlug(b *testing.B) {
	impls := []struct {
		name string
		f    func(string) string
	}{{"concat", SlugConcat}, {"builder", SlugBuilder}}
	for _, n := range []int{16, 256, 4096} {
		in := input(n)
		for _, im := range impls {
			b.Run(fmt.Sprintf("impl=%s/n=%d", im.name, n), func(b *testing.B) {
				b.ReportAllocs()
				b.SetBytes(int64(n))
				for b.Loop() {
					sink = im.f(in)
				}
			})
		}
	}
}

Reading it from the top:

• The file name ends in _test.go and the function in BenchmarkXxx(b *testing.B). That is all the go test tool needs.
• b.Run makes sub-benchmarks. The name impl=builder/n=256 is a convention that benchstat understands: segments of the form key=value become columns and rows, and the -col /impl flag in the script puts the implementations side by side per n.
• Inputs are built outside the loop. The string for each size is made once before b.Run, so its cost is not measured.
• b.ReportAllocs() and b.SetBytes add allocation counts and a throughput column. go test -benchmem does the first for the whole run; calling it in the benchmark makes it permanent for that benchmark.
• for b.Loop() runs the body as many times as needed to reach the benchmark time. It replaced the older for i := 0; i < b.N; i++ in Go 1.24 and does three useful things: the timer starts at the first call and stops at the last, so setup before the loop is not counted and b.ResetTimer() is not needed; the compiler cannot delete the call or hoist it out of the loop, because the body is kept alive; and the loop's count is decided by the framework without b.N leaking into your code.
• The result goes to a package variable (sink). b.Loop keeps the call alive, but the habit of assigning to a global survives older Go versions and keeps the code honest if someone changes it back to b.N.

Running it so the result means something

set -e
GOBIN=$PWD/bin go install golang.org/x/perf/cmd/[email protected] >/dev/null 2>&1
go test -run TestSame -count=1 .
go test -run '^$' -bench . -benchtime=200ms -count=6 . > bench.txt
echo "--- raw output, first 6 lines"
sed -n 1,10p bench.txt
echo
echo "--- benchstat, comparing the two implementations"
./bin/benchstat -col /impl bench.txt
rm -rf bin bench.txt

ok  	example.test/lab	0.386s
--- raw output, first 6 lines
goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
BenchmarkSlug/impl=concat/n=16-10         	  442006	       534.9 ns/op	  29.91 MB/s	     240 B/op	      29 allocs/op
BenchmarkSlug/impl=concat/n=16-10         	  480264	       505.7 ns/op	  31.64 MB/s	     240 B/op	      29 allocs/op
BenchmarkSlug/impl=concat/n=16-10         	  483663	       505.5 ns/op	  31.65 MB/s	     240 B/op	      29 allocs/op
BenchmarkSlug/impl=concat/n=16-10         	  485109	       502.3 ns/op	  31.86 MB/s	     240 B/op	      29 allocs/op
BenchmarkSlug/impl=concat/n=16-10         	  485980	       519.7 ns/op	  30.78 MB/s	     240 B/op	      29 allocs/op
BenchmarkSlug/impl=concat/n=16-10         	  470649	       503.6 ns/op	  31.77 MB/s	     240 B/op	      29 allocs/op

--- benchstat, comparing the two implementations
goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
               │    concat     │              builder               │
               │    sec/op     │   sec/op     vs base               │
Slug/n=16-10      505.60n ± 6%   44.94n ± 1%  -91.11% (p=0.002 n=6)
Slug/n=256-10    10680.5n ± 8%   575.9n ± 1%  -94.61% (p=0.002 n=6)
Slug/n=4096-10   924.109µ ± 4%   9.277µ ± 3%  -99.00% (p=0.002 n=6)
geomean            17.09µ        621.5n       -96.36%

               │    concat    │                 builder                 │
               │     B/s      │      B/s        vs base                 │
Slug/n=16-10     30.18Mi ± 5%    339.54Mi ± 1%  +1025.09% (p=0.002 n=6)
Slug/n=256-10    22.86Mi ± 8%    423.90Mi ± 1%  +1754.38% (p=0.002 n=6)
Slug/n=4096-10   4.225Mi ± 4%   421.071Mi ± 3%  +9866.70% (p=0.002 n=6)
geomean          14.28Mi          392.8Mi       +2649.87%

               │     concat      │               builder               │
               │      B/op       │     B/op      vs base               │
Slug/n=16-10         240.00 ± 0%     16.00 ± 0%  -93.33% (p=0.002 n=6)
Slug/n=256-10       32176.0 ± 0%     256.0 ± 0%  -99.20% (p=0.002 n=6)
Slug/n=4096-10   7814.732Ki ± 0%   4.000Ki ± 0%  -99.95% (p=0.002 n=6)
geomean             38.61Ki          256.0       -99.35%

               │    concat     │              builder              │
               │   allocs/op   │ allocs/op   vs base               │
Slug/n=16-10       29.000 ± 0%   1.000 ± 0%  -96.55% (p=0.002 n=6)
Slug/n=256-10     448.000 ± 0%   1.000 ± 0%  -99.78% (p=0.002 n=6)
Slug/n=4096-10   7139.000 ± 0%   1.000 ± 0%  -99.99% (p=0.002 n=6)
geomean             452.7        1.000       -99.78%

Five details in the script carry the method:

1. -run '^$' skips the tests, so only the benchmarks run, and a test that is slow does not eat the budget.
2. -benchtime=200ms shortens the default one second per benchmark, which is enough here because each iteration is microseconds. For code that costs milliseconds per call, raise it or use -benchtime=50x for a fixed count.
3. -count=6 runs every benchmark six times. This is the part people skip, and it is why the output ends in ± 6% and a p value.
4. The output goes to a file, so you can run the old code, save old.txt, change the code, save new.txt and compare them later with benchstat old.txt new.txt. Here the comparison is between two implementations in one run, using the /impl column.
5. benchstat is installed at a pinned version. Results from different tool versions can differ in how they compute intervals.

The raw output

The first six lines of bench.txt show the shape:

BenchmarkSlug/impl=concat/n=16-10   442006   534.9 ns/op   29.91 MB/s   240 B/op   29 allocs/op
BenchmarkSlug/impl=concat/n=16-10   480264   505.7 ns/op   31.64 MB/s   240 B/op   29 allocs/op
BenchmarkSlug/impl=concat/n=16-10   483663   505.5 ns/op   31.65 MB/s   240 B/op   29 allocs/op

The -10 suffix is GOMAXPROCS. The second column is the number of iterations the framework chose. ns/op is the cost of one call. The first sample (534.9) is above the other five, which sit between 502 and 520: the first iteration of a benchmark pays for cold caches and the processor still ramping up. Looking at one line would have given the wrong number for the fast case.

Reading benchstat

• The centre value is the median of the samples, not the mean, and the ± is the spread of the samples as a percentage of it. A change smaller than the spread is not a result.
• The p value is the probability of seeing this difference if the two groups were drawn from the same distribution. With six samples against six the lowest p the test can give is 0.002. benchstat marks differences that are not significant at the 0.05 level with a ~, and it prints no percentage for them. If you see ~, you have no evidence of a change, whatever the medians look like.
• The geomean row averages the ratios across the inputs and is useful for a summary, but the per size rows are the finding. A change that helps small inputs and hurts large ones disappears in a geomean.

The numbers make the quadratic nature of the first version visible. Going from 256 to 4096 bytes (16 times the input) multiplied the concat time by 86 (10,680 ns to 924,109 ns) and the builder time by 16 (575.9 ns to 9,277 ns). A benchmark at a single size would not have shown it.

The cause is in the other tables

The same run printed the allocation columns:

At 4096 bytes the string version made 7,139 allocations and 7.6 MiB of garbage to produce a 3 KB result, while the builder version made one. The ns/op difference is mostly the allocator and the collector. That is why -benchmem is worth leaving on: when a number changes, the allocation columns tell you whether the change came from doing less work or from allocating less.

Comparing a change against what you had

The lab compares two implementations in one run. The more common job is comparing the code before and after an edit. The workflow is mechanical, and keeping to it is what separates a measurement from an impression:

git stash          # or check out the old commit
go test -run '^$' -bench Slug -count=10 ./... > old.txt
git stash pop      # or check out the new commit
go test -run '^$' -bench Slug -count=10 ./... > new.txt
benchstat old.txt new.txt

Use ten samples when the machine is noisy, since the interval narrows with the count. If benchstat prints ~ next to a row, the edit did not move that benchmark by a measurable amount, and the right conclusion is that it made no difference, however much you wanted it to. Keep the two files with the commit message when the change is a performance fix, so the claim can be checked later.

For a change that touches a hot path in a service, add one more step: run the benchmark with the same GOMAXPROCS as production (-cpu 4, for a four core container), because contention that does not exist on ten cores can dominate on four.

Benchmarks that use more than one goroutine

b.Loop measures one goroutine calling the function. For code that is shared by many, such as a cache, a pool or a counter, use b.RunParallel, which starts GOMAXPROCS goroutines and reports the wall time per operation across all of them:

func BenchmarkCacheGet(b *testing.B) {
	c := newCache()
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			sink, _ = c.Get("key")
		}
	})
}

Read the result with care: ns/op there is total time divided by total operations, so ten goroutines doing 10 ns of work each in parallel report 1 ns. The number answers the question "how many operations per second can this structure serve", which is the right question for a shared structure and the wrong one for single call latency (sync.Pool and false sharing both use it).

Setup that must not be timed

With b.Loop the work before the loop and after it is excluded. Two situations still need care:

• Setup that must happen on every iteration (a fresh buffer, a reset data structure) belongs inside the loop and is therefore timed. Call b.StopTimer() and b.StartTimer() around it only if the setup is large, since stopping and starting the timer has its own cost and can make a nanosecond benchmark worse than the thing it measures. Often the better answer is to restructure: allocate the setup once and reset it cheaply.
• Expensive one time setup (loading a file, building a 100 MB index) goes before the loop, or into a helper that runs once and is shared by the sub-benchmarks.

If the setup is so large that the framework's first run takes seconds, give the benchmark a fixed count with -benchtime=20x instead of a duration.

When the numbers will not settle

A spread above about 5 percent on a microbenchmark usually has a cause you can fix:

1. Other work on the machine. Check the load average and quit browsers and builds. A benchmark that varies by 30 percent at noon and 2 percent at midnight is a warning about the host.
2. Power management. A laptop on battery, or a chip that throttles when hot, changes speed during the run. Plug in, and let the machine cool between long runs.
3. A benchmark that allocates a lot. The collector runs during the measurement and its timing varies. Run with GOGC=off for the experiment if you want to isolate the code, but report the default numbers too, since production collects.
4. Too short a run. With a few thousand iterations the first, cold ones weigh heavily. Raise -benchtime.
5. Non deterministic input. Fix the seed of any random data. A benchmark that sorts different data each run is measuring the data.

A regression gate in CI

Benchmarks are most useful as a gate on the code that matters most. Keep a short list (the parse path, the router, the encoder), run them on the same class of runner with -count=10, store the previous result as an artifact, and fail the build when benchstat reports a slowdown beyond a threshold with p < 0.01. A hosted runner is noisy, so set the threshold wide (20 percent is typical) and treat a failure as a prompt to rerun locally. A gate that fails on noise gets turned off, and a gate that never fails is worth nothing.

Habits that keep a benchmark honest

• Benchmark one thing. If the function reads a file, parses it and formats the result, benchmark the parse with the bytes already in memory.
• Use realistic inputs, in several sizes. The 16 byte case gave a 91 percent saving and the 4096 byte case 99 percent, and the right answer to "how much faster" depends on which your service sees.
• Check the answers match with a test in the same file, as TestSame does, before comparing speed.
• Keep the machine quiet. Close other heavy programs, plug in a laptop, and look at the load average. The lab ran with a load average near 3 on 10 cores. A noisy host shows up as a large ±.
• Run old and new in the same session, alternating if you can, since thermal throttling and background tasks drift over an hour.
• Do not compare across machines or Go versions unless the question is about the machine or the version.
• Profile from the benchmark when you need to know why: -cpuprofile cpu.out -memprofile mem.out write profiles of exactly the code under test (CPU profiles).

Checklist

• One BenchmarkXxx(b *testing.B) per question, using for b.Loop().
• Sub-benchmarks named key=value for each input size and implementation.
• ReportAllocs on, a test proving equal answers, inputs built before the loop.
• -run '^$' -count=6 and the output saved to a file.
• benchstat with a pinned version; trust a change only if it has a percentage and a p value under 0.05.
• Read the allocation columns to learn why a number moved.

The testing package is documented in testing.B, and the statistics in the benchstat documentation.