Seeded Random Tests in Go and Reproducible Failures

Random inputs find bugs that chosen inputs miss, and a random test that fails once and never again is worthless. The trade is simple to settle: let the test choose a seed, print it, and let the next run override it. The failure carries its own reproduction.

package main

import "math/rand/v2"

// Shuffle returns a shuffled copy.
func Shuffle(xs []int, r *rand.Rand) []int {
	out := append([]int(nil), xs...)
	r.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] })
	return out
}

func main() {}

package main

import (
	"math/rand/v2"
	"os"
	"strconv"
	"testing"
	"time"
)

// seed returns a reproducible seed and logs it, so a failure can be replayed with TEST_SEED.
func seed(t *testing.T) uint64 {
	t.Helper()
	s := uint64(time.Now().UnixNano())
	if v := os.Getenv("TEST_SEED"); v != "" {
		s, _ = strconv.ParseUint(v, 10, 64)
	}
	t.Logf("seed=%d (rerun with TEST_SEED=%d)", s, s)
	return s
}

func TestShuffleKeepsElements(t *testing.T) {
	r := rand.New(rand.NewPCG(seed(t), 0))
	out := Shuffle([]int{1, 2, 3, 4, 5}, r)
	t.Logf("shuffled: %v", out)
	sum := 0
	for _, v := range out {
		sum += v
	}
	if sum != 15 {
		t.Fatalf("elements changed: %v", out)
	}
}

F='s/ \([0-9.]+s\)//; s#^ +[a-z_]+\.go:[0-9]+: #    #'
echo "# two runs with random seeds"
for i in 1 2; do go test -count=1 -v . 2>&1 | grep -E 'shuffled' | sed -E "$F"; done
echo "# replay a seed"
for i in 1 2; do TEST_SEED=42 go test -count=1 -v . 2>&1 | grep -E 'seed=|shuffled' | sed -E "$F"; done

# two runs with random seeds
    shuffled: [1 2 4 5 3]
    shuffled: [5 2 1 4 3]
# replay a seed
    seed=42 (rerun with TEST_SEED=42)
    shuffled: [3 2 4 1 5]
    seed=42 (rerun with TEST_SEED=42)
    shuffled: [3 2 4 1 5]

The pattern

The seed helper picks a seed from the clock, unless the TEST_SEED environment variable is set, logs it in a form that can be pasted into a command, and returns it. The test builds its generator from the seed with rand.NewPCG(seed, 0). The first block of the output shows two runs with different shuffles. The second shows two runs with TEST_SEED=42, and the shuffle is the same both times, [3 2 4 1 5].

In CI, t.Logf output appears when a test fails, so the line seed=... (rerun with TEST_SEED=...) is in the failure log. Copy the number, run TEST_SEED=<n> go test -run TestName, and the failing input is back.

When it fails in CI

The sequence is short. A run fails, the log contains seed=1730032151234567000 (rerun with TEST_SEED=1730032151234567000), and you run that command on your machine. The input is the same, so the failure is the same, and you can add a print, attach a debugger or bisect the code. If it does not reproduce, the test depends on something other than the seed, and it is worth knowing that: a map iteration order, a real clock, shared state with another test. The seed has then done its second job, which is to prove the cause lies elsewhere.

Once you have fixed the bug, turn the failing input into a normal test case with fixed values, in a table, so it stays covered even if the random generator or the seed changes. The seed is a way to find the case, and the table is where it lives afterwards.

Details that matter

Use math/rand/v2. Its PCG and ChaCha8 sources have documented algorithms, so a recorded seed stays valid when Go upgrades, whereas the top-level functions are randomly seeded and cannot be replayed. Keep one generator per test, not a global one, so parallel tests do not consume each other's sequence and change what a seed produces.

Do not seed from the clock without printing the seed; that gives you the worst of both worlds, since the test is random and cannot be replayed. If the test uses a property library, it already does this: rapid prints -rapid.seed, as shown in the property based testing article. For a full treatment of the other sources of variation, see determinism. The generators are documented in the math/rand/v2 reference.