Test Concurrency Bugs With Deterministic Interleavings

Most concurrency tests run the code with many goroutines and hope. The bad schedule happens once in ten thousand runs, the test passes ninety-nine thousand times, and the bug ships. The alternative is to stop hoping and construct the schedule. If you know which interleaving breaks the code, you can force it, and then the test fails every time until the bug is fixed.

This article does that for a classic bug, check then act, on Go 1.25.6, and shows how the same test passes for the corrected code. The related tools are the race detector, which finds unsynchronised memory access, and property based testing, which can explore schedules; this is the precise tool for a bug you can already describe.

The bug

package main

import "sync"

// Account has a check-then-act bug: the balance is checked, then reduced, with no lock held between.
type Account struct {
	Balance    int
	afterCheck func() // test hook, called between the check and the update
}

func (a *Account) Withdraw(n int) bool {
	if a.Balance < n {
		return false
	}
	if a.afterCheck != nil {
		a.afterCheck()
	}
	a.Balance -= n
	return true
}

// SafeAccount holds the lock across the check and the update.
type SafeAccount struct {
	mu         sync.Mutex
	Balance    int
	afterCheck func()
}

func (a *SafeAccount) Withdraw(n int) bool {
	a.mu.Lock()
	defer a.mu.Unlock()
	if a.Balance < n {
		return false
	}
	if a.afterCheck != nil {
		a.afterCheck()
	}
	a.Balance -= n
	return true
}

func main() {}

Account.Withdraw checks that the balance covers the amount, then subtracts it. Between those two lines a second goroutine can run the same check against the same balance. If two withdrawals of 80 hit a balance of 100, both checks pass, and the balance ends at -60. No individual line is wrong; the sequence is.

Notice what the race detector says about this: nothing, if each access is protected, and in many real versions of this bug the accesses are protected individually. A check-then-act bug is a logic race. The detector finds data races, meaning unsynchronised reads and writes, and cannot tell that two correctly locked operations should have been one.

SafeAccount fixes it by holding the lock across the check and the update. Both types have an afterCheck hook, a function field that is nil in production. It is called at the vulnerable point so a test can pause there.

Forcing the schedule

package main

import (
	"sync"
	"testing"
	"time"
)

// Two withdrawals of 80 from a balance of 100. The hook makes both pass the check before either updates.
func TestForcedInterleavingBuggy(t *testing.T) {
	var arrived sync.WaitGroup
	arrived.Add(2)
	gate := make(chan struct{})
	a := &Account{Balance: 100}
	a.afterCheck = func() {
		arrived.Done()
		<-gate // wait until the other goroutine has also passed the check
	}

	var done sync.WaitGroup
	results := make([]bool, 2)
	for i := range 2 {
		done.Add(1)
		go func() { defer done.Done(); results[i] = a.Withdraw(80) }()
	}
	arrived.Wait() // both are between the check and the update
	close(gate)
	done.Wait()
	t.Logf("results=%v balance=%d", results, a.Balance)
}

// Same schedule against the locked version: the second goroutine cannot pass the check, so it never reaches the hook.
func TestForcedInterleavingSafe(t *testing.T) {
	gate := make(chan struct{})
	entered := make(chan struct{}, 2)
	a := &SafeAccount{Balance: 100}
	a.afterCheck = func() { entered <- struct{}{}; <-gate }

	var done sync.WaitGroup
	results := make([]bool, 2)
	for i := range 2 {
		done.Add(1)
		go func() { defer done.Done(); results[i] = a.Withdraw(80) }()
	}
	<-entered // one goroutine is inside, holding the lock
	select {
	case <-entered:
		t.Error("a second goroutine got past the check")
	case <-time.After(100 * time.Millisecond):
		t.Log("second goroutine is blocked on the lock, as it should be")
	}
	close(gate)
	done.Wait()
	t.Logf("results=%v balance=%d", results, a.Balance)
}

go test -count=1 -v -run 'Buggy|Safe' . 2>&1 | grep -vE '^(=== RUN|PASS|ok)' | sed -E 's/ \([0-9.]+s\)//; s#^ +[a-z_]+\.go:[0-9]+: #    #'

    results=[true true] balance=-60
--- PASS: TestForcedInterleavingBuggy
    second goroutine is blocked on the lock, as it should be
    results=[false true] balance=20
--- PASS: TestForcedInterleavingSafe

The buggy test builds a gate from two pieces of synchronisation. A WaitGroup with count 2 records how many goroutines have reached the hook, and a channel gate blocks them there. Each hook call does arrived.Done() and then waits on gate. The test starts two withdrawals, calls arrived.Wait(), which returns only when both are inside the hook, and closes gate. Both goroutines now resume with the check behind them, and both subtract. The log reads results=[true true] balance=-60. That outcome is not likely; it is certain, because the schedule that produces it is built into the test.

There are no sleeps. The wait is on an event, both arrived, and not on a guess about how long the goroutines take. That is what makes the test fast and what makes it reliable on a loaded machine.

The same schedule against the fix

The safe version needs a different arrangement, and the difference is the lesson. With the lock held across the check and the update, the second goroutine cannot get past the lock to reach the hook while the first is paused there. A test that waits for both to arrive would hang. So TestForcedInterleavingSafe waits for the first to enter, then checks that the second does not: a select on the entered channel with a 100 ms timer. The timer wins, and the log says second goroutine is blocked on the lock, as it should be. Then the gate opens, the first withdrawal completes and the second sees a balance of 20 and returns false. The results are [false true] or [true false] depending on which goroutine won, and the balance is 20 either way.

That timer is the one place a wait on absence appears. It is a bound on how long we wait to see that something does not happen, as discussed in the goroutines and channels article. It can give a false pass if the second goroutine is slow to start, but it cannot give a false failure, so the test never flakes. If you want certainty, count goroutines blocked on the mutex through runtime.Stack, or use testing/synctest, where synctest.Wait proves every goroutine in the bubble is blocked.

Where to put hooks

A hook is code that exists for tests. Keep it honest.

• Put it where the bug lives, between the two steps that must be atomic, and nowhere else. A hook per function turns the code into a testing framework.
• Make it a nil-checked function field, as here, or an interface with a no-op default. The production cost is one comparison.
• Do not export it. The field starts with a lower case letter, and tests in the same package set it. External callers cannot.
• Name it for the moment, afterCheck, beforeCommit, onRetry, so a reader of the production code sees what the test does there.

Some codebases prefer build tags to compile hooks out of release builds. That avoids the nil check and costs two files per hook; usually not worth it.

Other schedules worth forcing

The gate technique covers more than check-then-act.

• Lost update: two goroutines read, both modify, both write. Hook between read and write.
• Use after close: one goroutine closes a channel or connection while another is about to send. Hook before the send, close, release.
• Double initialisation: two callers of a lazy initialiser both see it as unset. Hook inside the check.
• Shutdown races: a request arrives after the server has begun to stop, as in graceful shutdown tests.
• Cancellation races: the context ends exactly between two steps. Hook between them and cancel.
• ABA problems in lock-free code: the value changes and returns to its original while a goroutine is suspended.

In each case you write down the bad schedule first, as a sequence of steps by named actors, and then each step becomes a gate. If you cannot write the schedule down, you do not yet understand the bug, and writing it is part of understanding.

The limits of the technique

Forced interleavings test the schedule you thought of. They find the bug you already suspect, not the one you do not. Use random exploration for discovery, with -race -count=100 and property tests, and forced schedules for proof: after you find a race in production, build the schedule that reproduces it, watch the test fail, fix, and keep the test. That test is a permanent guard against the regression, and unlike the flaky reproduction it will not be disabled for annoying people.

The standard library has no tool that enumerates schedules for you, so the hook and gate approach is what you can use today without a dependency. The Go memory model defines which orderings are guaranteed, and the sync package documentation covers the primitives used for the gate.