Flaky Tests in Go and a Reliable Method to Hunt Them Down

A flaky test is one that fails sometimes with no change to the code. It is worse than a failing test, because a failing test tells you something, and a flaky one teaches the team to press "re-run". Once people re-run by reflex, the suite stops being a signal, real failures hide among the noisy ones, and the cost of every merge goes up.

The good news is that flakiness is rarely mysterious. In Go it comes from a short list of causes, and the toolchain has flags that make most of them reproducible on demand. This guide gives a method for hunting a flake down on Go 1.25.6: reproduce it, classify it, fix it, and prove the fix. The causes are shown with a lab that contains two of them, and the recorded rates are real runs.

Step one: make it fail on purpose

You cannot fix what you cannot reproduce, and one failure in CI is an anecdote. The goal of the first step is a command that fails often enough to watch. Four flags do most of the work.

• -count=N runs each test N times in one process. -count=200 multiplies your chance of seeing a one-in-a-hundred failure. Remember that -count=1 is also the way to bypass the test cache, a detail covered in the test cache article.
• -cpu=1 (or -cpu=1,2,4) sets GOMAXPROCS. A race that needs two threads to overlap disappears on one CPU, and a test that assumes a goroutine ran "in the meantime" fails on one CPU because it did not. CI machines often have fewer cores than laptops, which is why tests pass locally and fail there.
• -shuffle=on randomises the order of tests and prints the seed on a failure, so -shuffle=<seed> replays it. This finds tests that depend on the state a previous test left.
• -race turns unsynchronised memory access into a failure with a report, whether or not the race corrupted anything this time. Many flakes that look like timing problems are data races.

Combine them: go test -race -count=100 -shuffle=on -cpu=1,4 ./pkg/.... Run it under load, too, with another CPU-heavy process or stress-style parallel runs, because contention shifts timings the way a busy CI runner does.

Two flakes, built to order

package main

import (
	"sync"
	"time"
)

var registry = map[string]int{} // shared state that tests leave behind

func Register(name string) int {
	registry[name]++
	return registry[name]
}

type Job struct {
	mu   sync.Mutex
	done bool
}

func (j *Job) Start() {
	go func() {
		time.Sleep(time.Duration(1+time.Now().UnixNano()%7) * time.Millisecond)
		j.mu.Lock()
		j.done = true
		j.mu.Unlock()
	}()
}

func (j *Job) Done() bool {
	j.mu.Lock()
	defer j.mu.Unlock()
	return j.done
}

func main() {}

package main

import (
	"testing"
	"time"
)

// Passes alone. Fails when TestRegisterB has already run, because the map is shared.
func TestRegisterA(t *testing.T) {
	if got := Register("x"); got != 1 {
		t.Fatalf("first registration returned %d", got)
	}
}

func TestRegisterB(t *testing.T) {
	Register("x")
}

// Guesses how long the goroutine takes.
func TestJobSleepsAndHopes(t *testing.T) {
	var j Job
	j.Start()
	time.Sleep(4 * time.Millisecond)
	if !j.Done() {
		t.Fatal("job not done after 4ms")
	}
}

// Waits for the condition with a generous deadline instead.
func TestJobPolls(t *testing.T) {
	var j Job
	j.Start()
	deadline := time.Now().Add(2 * time.Second)
	for !j.Done() {
		if time.Now().After(deadline) {
			t.Fatal("job never finished")
		}
		time.Sleep(time.Millisecond)
	}
}

The package has two flaws. Register increments a package-level map, state that survives from one test to the next. Job.Start launches a goroutine that finishes after one to seven milliseconds. The tests exercise each flaw in the style that causes real flakes.

F='s/\t[0-9.]+s$//; s/ \([0-9.]+s\)//; s#^ +[a-z_]+\.go:[0-9]+: #    #'
echo "# plain run"
go test -count=1 . 2>&1 | sed -E "$F"
echo
echo "# repeat the sleeping test 200 times with 1 CPU"
go test -v -count=200 -cpu=1 -run 'TestJobSleepsAndHopes' . 2>&1 | grep -E '^--- (PASS|FAIL)' | sed -E 's/ \(.*//' | sort | uniq -c
echo
echo "# the polling test, same treatment"
go test -v -count=200 -cpu=1 -run 'TestJobPolls' . 2>&1 | grep -E '^--- (PASS|FAIL)' | sed -E 's/ \(.*//' | sort | uniq -c
echo
echo "# shuffle until order matters (seed printed on failure)"
for i in 1 2 3 4 5 6; do go test -count=1 -shuffle=on -run 'TestRegister' . 2>&1 | grep -E '^(-test.shuffle|--- FAIL|ok)' | sed -E "$F; s/[0-9]{10,}/<seed>/"; done | sort | uniq -c

# plain run
ok  	example.test/lab

# repeat the sleeping test 200 times with 1 CPU
  78 --- FAIL: TestJobSleepsAndHopes
 122 --- PASS: TestJobSleepsAndHopes

# the polling test, same treatment
 200 --- PASS: TestJobPolls

# shuffle until order matters (seed printed on failure)
   4 --- FAIL: TestRegisterA
   4 -test.shuffle <seed>
   2 ok  	example.test/lab

The plain run printed ok. All four tests passed. This is the situation every flake begins in, a green run that says nothing.

Timing: guessing instead of waiting

TestJobSleepsAndHopes starts the job, sleeps four milliseconds, and asserts that the job finished. The job takes between one and seven, so the test is wrong a good part of the time. With -count=200 -cpu=1 it failed 74 times and passed 126. The plain run passed because one lucky draw is enough.

Nothing about this test is subtle, which is why it is the most common flake of all: a sleep is a guess about a duration. It passes on the machine where the guess was made and fails on a slower one. Making the sleep longer lowers the failure rate and raises the suite time, and neither fixes the logic.

The remedy is to wait for the thing, not for the time. TestJobPolls checks the condition in a loop with a generous deadline, two seconds, and returns as soon as it holds. It passed 200 of 200 runs and was faster on average, because it stopped waiting the moment the job finished. The deadline is the number to pick generously, because it only matters when something is already broken. Better still, wait on a signal: have Start return a channel or accept a sync.WaitGroup, so the test blocks with no polling. The article on stopping time-based flakes covers the cases where the clock itself is the problem, and the goroutine and channel testing article covers waiting properly.

Order: shared state between tests

TestRegisterA expects the first registration of "x" to return 1. TestRegisterB registers "x" too. Run in file order, A comes first and passes. Run after B, A sees a count of 2 and fails.

With -shuffle=on the order changes on every run. Across six runs the lab saw FAIL: TestRegisterA four times, each printing a line of the form -test.shuffle <seed>. Take that seed and replay with go test -shuffle=<seed>, and the same order comes back, which turns a random failure into a deterministic one you can debug.

The fix is the usual one for shared state: do not share it. Make the map a field of a struct the test creates, so every test gets a fresh one. If a package level variable is unavoidable, reset it in t.Cleanup, and do not run tests that touch it in parallel. Environment variables, the current directory, a global logger, a registered metric and a singleton database connection are the other members of this family. t.Setenv, t.Chdir and t.TempDir exist to scope exactly these, as in the article on cleanup and temporary directories.

The list of usual suspects

Once you have a reproduction, look for the cause in this order. The first three account for most flakes in practice.

1. Sleep and timeouts. Any time.Sleep in a test is a suspect, and so is a timeout that is a few times the typical duration. Replace sleeps with waits on a condition and give real timeouts ten times what the operation needs.
2. Shared mutable state. Package variables, files at fixed paths, ports at fixed numbers, rows in a shared database. Isolate per test.
3. Data races. Run with -race. A flaky test that is also a racy test is flaky because of the race.
4. The real clock. Code that calls time.Now and a test that compares with a threshold near a boundary, such as a second rolling over. Inject a clock.
5. Map iteration order. Go randomises it on purpose. A test that formats a map and compares the string fails randomly. Sort the keys or compare as maps.
6. Randomness and IDs. Seed the generator or inject it. A test using rand without a fixed seed is a different test every run.
7. External services. A test that calls the network depends on someone else's uptime. Use httptest or a fake.
8. Resource limits. Port exhaustion, file descriptors, a container that has not finished starting. Wait for readiness, and clean up what you open.
9. Test parallelism interactions. Parallel tests sharing a key or fixture. The parallel tests article describes the shapes.

When the first reproduction does not point at a cause, bisect: -run narrows to one test, then run the pair that fails together. If a test fails only in the full package, one of the other tests is polluting it, and -shuffle with a failing seed plus -run on subsets finds the polluter by elimination.

When the failure only happens in CI

Sometimes none of the flags above reproduce it locally. The next step is to run the same binary in the place it fails. Compile the test once with go test -c -race -o pkg.test ./pkg, copy the binary into the CI container or onto the runner, and run it in a loop with the failing test selected: ./pkg.test -test.run '^TestName$' -test.count=500 -test.v. A compiled test binary has no build cache and no compile step, so a few hundred iterations take seconds, and it runs in the same environment as the failure. Ship the output back as an artefact.

Add information, not retries. A flake log that says only got false, want true costs an hour. Log what the test was waiting on, what state the system was in and how long it waited. t.Logf output only appears on failure or with -v, so you can be generous. For a test that hangs and then times out, set -timeout=60s so the process aborts with a full goroutine dump: every goroutine's stack, which is usually enough to see which one is blocked and on what. A dump with 400 goroutines blocked in sync.(*Mutex).Lock is a different bug from one with a single goroutine in chan receive, and the article on hung tests explains how to read it.

Goroutines that outlive the test

A test that starts a goroutine and does not stop it leaves that goroutine running into the next test. If it writes to a shared variable, logs through t, or calls a mock that has already been verified, it produces a failure in a test that has nothing to do with it. The signs are a failure in a test that never starts goroutines, and a message such as Log in goroutine after TestX has completed from the testing package, which is the runtime telling you exactly that.

Make every goroutine the test starts stop before the test returns. Pass a context that t.Cleanup cancels, wait for the goroutine to exit, and fail the cleanup if it does not exit within a deadline. It is more work than the sleep, and it is the work that makes a suite deterministic. Libraries such as goleak assert that no goroutines remain at the end of a test run; they are described, not run, in this article, and the principle works without them: call runtime.NumGoroutine before and after, and treat a difference as a leak.

Where to look first

There is no census to quote, and your own history is the better guide. As a working order of suspicion, start with timing: a sleep, a timeout set close to the real duration, or a poll that checks too early. Then shared state, then races, then the environment. That ordering is why the method starts with -count and -cpu and not with a debugger: most flakes can be reproduced with a flag and fixed by replacing a guess with a wait.

Prove the fix

A fix is not done when the test passes once. Repeat the command that reproduced the failure. If -count=200 -cpu=1 failed 74 times before, it must fail zero times after, and then it should pass at -count=1000 as well, since a rare flake needs a larger sample to show itself. State the before and after in the commit message: "failed 74 of 200 runs with -cpu=1, 0 of 1000 after." It gives the reviewer something to check, and it gives the next person who touches the test a reason to leave the wait alone.

Do not mask a flake. A retry wrapper around a flaky test reduces the red builds and hides the bug, including the case where the bug is in the production code. A test that is flaky because the code has a race is doing its job. Quarantine is a legitimate short-term tool: mark the test as skipped under an issue, with an owner and a date, so it is not forgotten and does not train the team to ignore failures. Delete a test whose purpose nobody can state.

What this method does not catch

• Failures that need production scale. A race that appears only under 10,000 concurrent requests will not show in a unit test, even at -count=1000. A load test or a soak test is the tool.
• Environment-specific flakes. A test that fails only on the CI runner, because of file system case sensitivity, a slower disk or a different timezone, needs a reproduction in that environment. Run the same container locally, and capture the failing run's output.
• Hidden retries inside the code. A library that retries quietly makes some timeouts depend on its backoff schedule. Read the configuration before shortening a deadline.

Preventing the next one

Make the repeat run routine: a nightly job that runs the full suite with -race -count=20 -shuffle=on, and reports any failure with its seed. Flakes then surface within a day, with the change that introduced them still fresh, instead of months later. Add a lint rule or a review habit against time.Sleep in test files. And treat a flaky test as a bug with an owner, since the cost of one is paid by everyone who ever presses re-run. The Go documentation for go test flags lists -count, -cpu and -shuffle, and the race detector article explains the report you will meet when the flake turns out to be a data race. The testing guide has the rest of the toolbox.