Test Concurrent Code in Go Without Flaky Failures or Sleeps

A flaky test is a test that has learned to lie. It passes on the author's laptop, fails once in fifty runs on CI, gets retried, and eventually gets ignored, which means the code it covered has no test. Tests of concurrent code are the main source of flakiness, and almost always for the same reason: the test makes an assumption about timing. "The goroutine will have finished by now." "The message will have arrived after 10 ms." Nothing in the language backs those assumptions up, and a busy machine breaks them.

The cure is to remove the assumption instead of making the number bigger. This guide covers the techniques in the order I reach for them: wait for an event, inject the sources of time, bound every wait, poll only when forced, and run the suite in ways that shake out bad interleavings. It builds on data races, the race detector and goroutine leaks, and sits in the concurrency patterns cluster. The sequel for time-based code is deterministic time tests with synctest. Everything ran on Go 1.25.6.

Why sleep fails

A test that starts a goroutine and then sleeps before checking its effect encodes a race between the goroutine's work and the test's timer. When the work is fast, the test passes. When the machine is loaded, the work is slower than the sleep, and the test fails. The failure rate depends on the environment, so the bug is invisible locally and surfaces in CI, usually on a Monday.

The sleep also costs time when it works. A suite with two hundred tests that each sleep for 100 ms to be safe takes twenty seconds of doing nothing. People then lower the sleeps to save time, which raises the failure rate, and the cycle continues.

This program models the effect. A worker takes between 0 and 3 ms (a seeded random delay, standing in for scheduler and I/O jitter), and two checks look at the result: one that sleeps 2 ms and then reads it, and one that waits for the worker to signal completion:

package main

import (
	"fmt"
	"math/rand/v2"
	"sync/atomic"
	"time"
)

type worker struct {
	result atomic.Int64
	done   chan struct{}
}

func start(delay time.Duration) *worker {
	w := &worker{done: make(chan struct{})}
	go func() {
		time.Sleep(delay) // variable time, as on a real machine
		w.result.Store(42)
		close(w.done)
	}()
	return w
}

func main() {
	r := rand.New(rand.NewPCG(1, 2))
	sleepFailed, signalFailed := 0, 0
	const runs = 300

	for range runs {
		w := start(time.Duration(r.IntN(3000)) * time.Microsecond)
		time.Sleep(2 * time.Millisecond) // "should be enough"
		if w.result.Load() != 42 {
			sleepFailed++
		}
		<-w.done
	}
	for range runs {
		w := start(time.Duration(r.IntN(3000)) * time.Microsecond)
		select {
		case <-w.done:
			if w.result.Load() != 42 {
				signalFailed++
			}
		case <-time.After(5 * time.Second):
			signalFailed++
		}
	}
	fmt.Printf("sleep-based check failed %d of %d runs; signal-based check failed %d of %d\n",
		sleepFailed, runs, signalFailed, runs)
}

sleep-based check failed 92 of 300 runs; signal-based check failed 0 of 300

The sleeping check failed in 92 of 300 runs, roughly the share of delays longer than 2 ms, and the signal-based check never failed. That is a model of jitter rather than a measurement of your CI, but the lesson transfers: a sleep is correct only when the sleep exceeds the work, and the work is the part you do not control. Waiting for the signal is correct whatever the delay, and it is also faster on average, since it returns when the work is done instead of at a fixed time.

Wait for an event

The first rule is that the code under test should give the test something to wait for. Most concurrent code already has such a thing, because the real caller needs it too: a Done channel, a WaitGroup, a result channel, a callback. If it does not, add the smallest hook that makes it observable. The example for the rest of this article is a batcher: items are added, and the batch is flushed when it reaches a size or when a ticker fires, whichever comes first. The ticker is an input, a <-chan time.Time, not something the batcher creates for itself. That choice makes it testable:

package main

import "time"

// Batcher collects items and passes them to flush in groups.
type Batcher struct {
	in    chan string
	done  chan struct{}
	size  int
	tick  <-chan time.Time
	flush func([]string)
}

// NewBatcher flushes when size items are waiting or when tick fires, and once more on Close.
func NewBatcher(size int, tick <-chan time.Time, flush func([]string)) *Batcher {
	b := &Batcher{in: make(chan string), done: make(chan struct{}), size: size, tick: tick, flush: flush}
	go b.loop()
	return b
}

func (b *Batcher) Add(item string) { b.in <- item }

// Close flushes what is left and returns after the loop has exited.
func (b *Batcher) Close() {
	close(b.in)
	<-b.done
}

func (b *Batcher) loop() {
	defer close(b.done)
	var batch []string
	emit := func() {
		if len(batch) > 0 {
			b.flush(batch)
			batch = nil
		}
	}
	for {
		select {
		case item, ok := <-b.in:
			if !ok {
				emit()
				return
			}
			batch = append(batch, item)
			if len(batch) >= b.size {
				emit()
			}
		case <-b.tick:
			emit()
		}
	}
}

In production, main passes time.NewTicker(time.Second).C. In a test, the test passes a channel it owns and sends on it when it wants a tick. The flush callback is the observable event: the test makes it send the batch on a channel, so the test waits for exactly what it cares about.

package main

import (
	"reflect"
	"runtime"
	"testing"
	"time"
)

// recv waits for a batch with a deadline, so a bug fails the test instead of hanging it.
func recv(t *testing.T, flushed <-chan []string) []string {
	t.Helper()
	select {
	case batch := <-flushed:
		return batch
	case <-time.After(2 * time.Second):
		t.Fatal("no flush within 2s")
		return nil
	}
}

func newTestBatcher(size int) (*Batcher, chan time.Time, <-chan []string) {
	tick := make(chan time.Time)
	flushed := make(chan []string, 10)
	b := NewBatcher(size, tick, func(batch []string) {
		flushed <- append([]string(nil), batch...) // copy: the batcher reuses its slice
	})
	return b, tick, flushed
}

func TestFlushesWhenFull(t *testing.T) {
	b, _, flushed := newTestBatcher(3)
	defer b.Close()
	b.Add("a")
	b.Add("b")
	b.Add("c")
	if got := recv(t, flushed); !reflect.DeepEqual(got, []string{"a", "b", "c"}) {
		t.Fatalf("got %v", got)
	}
}

func TestFlushesOnTick(t *testing.T) {
	b, tick, flushed := newTestBatcher(100)
	defer b.Close()
	b.Add("only")
	tick <- time.Now() // the test decides when a second has passed
	if got := recv(t, flushed); !reflect.DeepEqual(got, []string{"only"}) {
		t.Fatalf("got %v", got)
	}
}

func TestCloseFlushesTheRemainder(t *testing.T) {
	b, _, flushed := newTestBatcher(100)
	b.Add("left")
	b.Close()
	if got := recv(t, flushed); !reflect.DeepEqual(got, []string{"left"}) {
		t.Fatalf("got %v", got)
	}
}

func TestCloseStopsTheGoroutine(t *testing.T) {
	before := runtime.NumGoroutine()
	b, _, _ := newTestBatcher(1)
	b.Close() // returns only after the loop exits
	deadline := time.Now().Add(time.Second)
	for runtime.NumGoroutine() > before && time.Now().Before(deadline) {
		time.Sleep(time.Millisecond)
	}
	if got := runtime.NumGoroutine(); got > before {
		t.Fatalf("goroutines: %d before, %d after Close", before, got)
	}
}

=== RUN   TestFlushesWhenFull
--- PASS: TestFlushesWhenFull (0.00s)
=== RUN   TestFlushesOnTick
--- PASS: TestFlushesOnTick (0.00s)
=== RUN   TestCloseFlushesTheRemainder
--- PASS: TestCloseFlushesTheRemainder (0.00s)
=== RUN   TestCloseStopsTheGoroutine
--- PASS: TestCloseStopsTheGoroutine (0.00s)
PASS
ok  	example.test/lab	1.248s

The tests run in microseconds because nothing in them waits for real time. TestFlushesOnTick sends on the channel it handed to the batcher, so "one second passes" is a single send. If the batcher's loop is broken and never flushes, the test fails after two seconds with "no flush within 2s", which is a diagnosable failure. Without the deadline it would hang until the test timeout kills the whole package and leaves no hint about which test was stuck. The pattern generalizes: inject every source of time and every source of events as an input, and the test becomes a script of sends and receives.

Bound every wait

The recv helper is the second rule in miniature. Any wait in a test, on a channel, a WaitGroup or a lock, must have a deadline. A WaitGroup.Wait has none built in, so wrap it:

func waitTimeout(t *testing.T, wg *sync.WaitGroup, d time.Duration) {
	t.Helper()
	done := make(chan struct{})
	go func() { wg.Wait(); close(done) }()
	select {
	case <-done:
	case <-time.After(d):
		t.Fatalf("timed out after %v waiting for goroutines", d)
	}
}

Choose a generous deadline (seconds), because it only matters when the test is already failing. A deadline is not an assumption about speed; a passing test never waits for it. Compare that with a sleep, where the wait is always paid in full. go test -timeout is a blunt version of the same idea for the whole package, and t.Context() (Go 1.24) gives a context cancelled when the test ends, which is a convenient parent for anything the test starts (context in tests).

Polling, as a last resort

Sometimes the code gives you nothing to wait on. A background refresh updates a field, a metric is incremented by a library, an external process writes a file. The test must look until the condition holds. The honest version polls with a short interval and a long deadline:

func eventually(t *testing.T, timeout time.Duration, cond func() bool, msg string) {
	t.Helper()
	deadline := time.Now().Add(timeout)
	for time.Now().Before(deadline) {
		if cond() {
			return
		}
		time.Sleep(time.Millisecond)
	}
	t.Fatalf("condition not met within %v: %s", timeout, msg)
}

The test returns as soon as the condition holds, so a fast machine pays one or two milliseconds. A slow machine has seconds. Polling is a smell, though, not a pattern to build on: it means the code under test cannot tell you what it is doing. If you control the code, add the hook instead.

Force an interleaving with a gate

Some properties are about what happens while something else is in progress: at most N tasks run at once, a second request waits for the first, cancellation interrupts work that is under way. A sleep cannot create those states reliably. A gate can: a fake task that announces it has started and then blocks until the test opens the gate. While the tasks are held at the gate, the test inspects the state, and then it lets them go. This test checks that a pool never runs more than three jobs at once and that it does reach three, without a single sleep:

package main

import "sync"

// Run executes the jobs with at most limit running at the same time.
func Run(limit int, jobs []func()) {
	queue := make(chan func())
	var wg sync.WaitGroup
	for range limit {
		wg.Go(func() {
			for job := range queue {
				job()
			}
		})
	}
	for _, j := range jobs {
		queue <- j
	}
	close(queue)
	wg.Wait()
}

package main

import (
	"sync/atomic"
	"testing"
	"time"
)

func TestRunNeverExceedsTheLimit(t *testing.T) {
	const limit, total = 3, 10
	var running, peak, finished atomic.Int32
	saturated := make(chan struct{}) // closed when `limit` jobs are inside at once
	gate := make(chan struct{})      // jobs wait here until the test opens it

	job := func() {
		n := running.Add(1)
		for {
			p := peak.Load()
			if n <= p || peak.CompareAndSwap(p, n) {
				break
			}
		}
		if n == limit {
			select {
			case <-saturated:
			default:
				close(saturated)
			}
		}
		<-gate // held here until released
		running.Add(-1)
		finished.Add(1)
	}

	jobs := make([]func(), total)
	for i := range jobs {
		jobs[i] = job
	}
	done := make(chan struct{})
	go func() { Run(limit, jobs); close(done) }()

	select {
	case <-saturated: // the pool really did reach its limit
	case <-time.After(2 * time.Second):
		t.Fatal("the pool never ran 3 jobs at once")
	}
	close(gate) // release every job
	select {
	case <-done:
	case <-time.After(2 * time.Second):
		t.Fatal("Run did not finish")
	}
	if peak.Load() > limit {
		t.Fatalf("peak concurrency %d exceeded the limit %d", peak.Load(), limit)
	}
	if finished.Load() != total {
		t.Fatalf("%d of %d jobs finished", finished.Load(), total)
	}
}

=== RUN   TestRunNeverExceedsTheLimit
--- PASS: TestRunNeverExceedsTheLimit (0.00s)
PASS
ok  	example.test/lab	1.247s

The gate holds the first three jobs inside the pool until the saturated channel proves all three are in at the same time. At that point the test knows the state exactly: three running, seven waiting, and the pool cannot start a fourth because the workers are all busy. Only then does it open the gate. Two assertions follow from facts, not from timing: the peak never exceeded the limit, and every job finished. If someone changes the pool to start a worker per job, the peak reaches ten and the test fails. The same shape works for "cancel while a request is in flight" (the fake handler signals entered, the test cancels, the handler returns ctx.Err()) and for "a second caller waits for the first" (the first holds the gate).

Run the suite so it can fail

A test that passes once proves little about concurrency. Three habits widen the net:

• -race on every run. It converts silent corruption into a failure with two stacks (the race detector).
• -count=N to repeat. A race that shows up one run in twenty shows up in -count=100 almost every time. I run -count=50 on packages with goroutines in CI nightly and -count=1 on every push.
• -cpu 1,2,8 to vary GOMAXPROCS. A bug that needs real parallelism hides at one CPU, and a bug that needs preemption hides at many.
• -shuffle=on to randomize test order and expose tests that depend on shared state from an earlier one.

t.Parallel() helps in a different way: it runs tests concurrently with each other, which surfaces global state that tests share by accident. It needs tests to be independent, so use it where that is true, and never as a way to make a flaky test "pass" by removing the contention that made it fail.

Clean up in t.Cleanup, not at the end of the test

A test that starts a goroutine owns it until the test ends, including when it fails halfway. If the stop call is the last line of the test, a t.Fatal earlier skips it and the goroutine keeps running into the next test, where it produces confusing failures: output from the wrong test, a port that is still bound, a race with the next test's setup. Register cleanup immediately after starting the thing, so it runs whichever way the test ends:

b, tick, flushed := newTestBatcher(3)
t.Cleanup(b.Close) // runs after the test returns, even after t.Fatal

t.Cleanup functions run last-in, first-out, which matches the order dependencies should be torn down in. For a test that starts a server, a client and a worker, registering each as it is created gives you the right shutdown order without writing it down.

What not to assert

Concurrent code gives you fewer guarantees than sequential code, and a test that asserts more than the code promises is flaky by construction. Do not assert the order in which two independent goroutines finish unless the code orders them. Collect results into a set or sort them before comparing. Do not assert exact timings; assert that something happened before a generous deadline, or use virtual time. Do not assert on the interleaving of log lines from different goroutines. And do not assert the absence of an event by waiting a little and seeing nothing: "no second flush happened" cannot be proven by a sleep. Prove it structurally, by closing the component and checking that the flushed channel yields exactly the expected batches, or with a gate that holds the state you want to inspect.

Check that goroutines stop

The last test above, TestCloseStopsTheGoroutine, is the cheapest leak check: compare the goroutine count before and after, allowing a short settle time. For packages with many goroutines, a leak checker in TestMain fails any test that leaves one behind (goleak, counting goroutines). A component that has Close or Stop should have a test that calls it and asserts that it returned and that the goroutines are gone.

What about time-based code

The injected ticker above handles code that takes a channel. Code that calls time.Now, time.Sleep or time.After directly is harder: the test either waits in real time or you wrap time behind an interface with a fake implementation (fake clocks). Since Go 1.25 the standard library has a third option, testing/synctest, which runs a test in a bubble where time is virtual and advances instantly whenever every goroutine is blocked, so unmodified code that sleeps for an hour tests in microseconds (synctest). Both are better than a real sleep. For new code I design for injection, and for existing code I reach for synctest.

Making the code easier to test

Most hard-to-test concurrent code is hard because it reaches for the outside world directly. The batcher above is easy to test because three decisions were made in its constructor: time comes in as a channel, the effect goes out as a callback, and stopping is a method that waits. Those three habits cover most cases. Take a clock or a ticker as a parameter. Report outcomes through a return value, a channel or a callback, and not only through a log line. Give every component a Close or Stop that returns when its goroutines have exited (starting and stopping goroutines). Code written that way tests quickly and deterministically; code that creates its own tickers, starts untracked goroutines and logs its results forces the test to sleep and hope.

A checklist for a flaky test

When a concurrent test flakes, I go down a list. Does it sleep? Replace the sleep with a signal. Does it wait without a deadline? Add one. Does it read shared state while goroutines may still write it? Wait for them first, and run it under -race. Does it depend on the order of two goroutines? Make the order explicit with a channel. Does it rely on real time? Inject the time. Does it leave goroutines running into the next test? Stop them and check. Nine times in ten the flake is one of these, and the fix makes the test faster as well as stable. The testing package documentation covers t.Context, t.Cleanup and parallel tests, and the Go blog's article on testing time and concurrency with synctest describes the newer option in detail.