Compare Benchmark Runs in Go With benchstat Properly

You change a function, run the benchmark before and after, and see 105 ns against 98 ns. Is that a 7% win? On a laptop the answer is usually "maybe", because the same code run twice can differ by more than that. The way to settle it is not a more careful look at two numbers. It is to collect many samples of each and ask a statistics tool whether the difference is bigger than the noise.

benchstat is that tool, maintained by the Go team in the golang.org/x/perf module. This article uses it on Go 1.25.6 with a change large enough to be obvious and a control that shows what "no change" looks like. The previous article on writing honest benchmarks covers the part before it, the test flags reference lists the -count and -benchtime options, and the b.Loop note covers the loop style used here.

One benchmark, two implementations

The comparison needs the same benchmark name in both files, which means the benchmark must run against old code and new code. A simple way in one package is a build tag: the old implementation behind //go:build old, the new behind its negation, and one benchmark that calls format.

//go:build old

package main

import "fmt"

func format(n int) string { return fmt.Sprintf("%d", n) }

//go:build !old

package main

import "strconv"

func format(n int) string { return strconv.Itoa(n) }

package main

func main() {}

package main

import "testing"

func BenchmarkFormat(b *testing.B) {
	for i := 0; b.Loop(); i++ {
		format(i)
	}
}

In a real project you would use git stash or two checkouts instead, and run the same command on each. The tag trick is for when you want both versions in one place, as in an article or a design spike. The only requirement is that the benchmark function has the same name in both runs.

Collect samples, then compare

go install golang.org/x/perf/cmd/[email protected] 2>&1 | grep -v '^go: downloading'
B='go test -run=^$ -bench=Format -benchmem -count=10'
$B -tags old . | grep -v '^PASS' > old.txt
$B . | grep -v '^PASS' > new.txt
$B . | grep -v '^PASS' > new2.txt
echo "# old (Sprintf) against new (Itoa)"
benchstat old.txt new.txt
echo
echo "# the same code against itself"
benchstat new.txt new2.txt

# old (Sprintf) against new (Itoa)
goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
          │   old.txt   │               new.txt               │
          │   sec/op    │   sec/op     vs base                │
Format-10   56.72n ± 4%   19.77n ± 3%  -65.15% (p=0.000 n=10)

          │   old.txt   │              new.txt               │
          │    B/op     │    B/op     vs base                │
Format-10   16.000 ± 0%   7.000 ± 0%  -56.25% (p=0.000 n=10)

          │  old.txt   │               new.txt               │
          │ allocs/op  │ allocs/op   vs base                 │
Format-10   2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)

# the same code against itself
goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
          │   new.txt   │           new2.txt            │
          │   sec/op    │   sec/op     vs base          │
Format-10   19.77n ± 3%   20.11n ± 2%  ~ (p=0.362 n=10)

          │  new.txt   │            new2.txt            │
          │    B/op    │    B/op     vs base            │
Format-10   7.000 ± 0%   7.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

          │  new.txt   │            new2.txt            │
          │ allocs/op  │ allocs/op   vs base            │
Format-10   0.000 ± 0%   0.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

The script installs benchstat at a pinned version, then runs the benchmark ten times per version. -count=10 repeats the whole benchmark ten times, producing ten lines, which are ten samples of the time. Ten is the practical minimum; with fewer the confidence interval is wide, and 20 is better when the noise is high. The grep -v '^PASS' drops the trailer so each file has only benchmark lines and the header.

benchstat old.txt new.txt prints one table per metric. Read the first.

Format-10   58.97n ± 4%   20.09n ± 2%  -65.93% (p=0.000 n=10)

• The value is the median across the samples, 58.97n meaning nanoseconds. Benchstat uses the median because one slow run, from a background process, would drag a mean.
• The ± 4% is the spread of the samples, so a wide spread tells you the machine was noisy.
• -65.93% is the change from the old column to the new one.
• p=0.000 is the p-value from a Mann-Whitney U test: the probability of seeing a difference this large if the two sets of samples came from the same distribution. Benchstat's default threshold is 0.05. Below it, the difference is reported; above it, a tilde appears.
• n=10 is the sample count on each side.

The new version is about three times faster, and that is not in doubt.

The other two tables are the memory columns. B/op went from 16 to 7 and allocs/op from 2 to 0, printed as -100.00%. A B/op that is not a whole number for a zero-allocation function is an averaging artefact of how the runtime counts small allocations, and the allocation count is the figure to trust. Sprintf boxes its argument into an any and builds the string through a buffer; Itoa formats small numbers without allocating at all.

The control that matters

The second comparison is the one to copy into your own process. It runs the new code twice and compares the result with itself: 20.09n ± 2% against 20.06n ± 1%, shown as ~ (p=0.669 n=10).

The tilde means "no significant difference". Those two files hold the same code, so the correct answer is no difference, and the p-value of 0.669 says so. This is your noise floor. When the control shows ~, a real change of a few percent will also show a number; when your control shows a p-value near 0.05 or a spread of 10%, a 5% improvement means nothing. Run a control at the start of any benchmarking session, especially on a laptop. A control that flags a difference is telling you the machine, and not the code, is the thing changing.

Be careful with the p-value from the other direction. At a threshold of 0.05, one comparison in twenty between identical code will show a difference by chance. If you compare fifty benchmarks in one run, expect some false positives and distrust a tiny effect in one of them. Look for a consistent direction and a size that makes sense given what you changed.

Reading a result you did not expect

When the numbers disagree with your expectations, check these before believing either.

1. The spread is large. ± 15% means the samples are scattered. Re-run with the machine idle, or increase -count.
2. The benchmark measures something else. A result near 0.3 ns/op is a loop the compiler removed, as described in the benchmarks guide.
3. Different environments between files. Benchstat prints the header lines (goos, cpu) and warns when two files disagree. Take both runs on the same machine, close in time, with the same power settings.
4. Setup is inside the loop. The cost of building the data is part of the number.
5. Thermal throttling. Run old, new, old, new in turns, not all old first, if the effect is small. The -count flag runs a benchmark repeatedly, but not interleaved with another file's, so a laptop that heats up penalises whichever ran last.

Benchstat also accepts more than two files, in which case it compares each to the first, and -col and -row options regroup the table by name or by a configuration key. The documentation in the benchstat package lists the options.

Using it in a pull request

The most useful place for benchstat is review. A change that claims to be faster should include the table: benchstat old.txt new.txt pasted into the description, with the hardware lines. That is a few lines, it is checkable, and it ends arguments about whether the optimisation is real.

For continuous tracking, store the benchmark output for each commit and compare the pull request with the base branch in CI. Treat that as a signal that triggers a look, not as a gate that fails a build: shared CI runners are noisy, and a gate with a 5% threshold on a noisy runner fails builds at random. If a gate is needed, run the old and new binaries on the same job, one after another, and set the threshold well above the control's spread.

Where benchstat does not help

It tells you that two sets of numbers differ. It does not tell you why, or whether the difference matters. A 65% improvement of a function that takes 0.1% of the request time is a 0.07% improvement of the request. It cannot repair a benchmark that measures the wrong thing, and it assumes the samples are independent. And it compares distributions of timing; it says nothing about correctness, so a faster function that returns a wrong answer wins. Pair every optimisation with its test, and keep the profile that pointed you at the function in the same description as the table.

The Go team documents the tool and its statistical method in the golang.org/x/perf repository. For the benchmark side of the pair, see the testing.B reference.