Run the Race Detector in Go Tests and in CI Pipelines

A data race is two goroutines touching the same memory at once, with at least one of them writing and nothing ordering the accesses. The result is not a crash. It is a counter that is a little low, a map that sometimes panics, a struct read half-updated. Those failures are rare, depend on timing, and vanish when you add a log line. The race detector exists because that kind of bug cannot be found by reading test output.

The detector is a flag on go test, and the question for a team is not how to turn it on, but how to make it a habit. This article shows what it reports on Go 1.25.6, why the plain test passes, and how to run it in CI without making the pipeline miserable.

A test that passes

package main

import "sync"

// Counter is not safe for concurrent use.
type Counter struct{ n int }

func (c *Counter) Inc() { c.n++ }

// SafeCounter is.
type SafeCounter struct {
	mu sync.Mutex
	n  int
}

func (c *SafeCounter) Inc() {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.n++
}

func main() {}

package main

import (
	"sync"
	"testing"
)

func hammer(inc func()) {
	var wg sync.WaitGroup
	for range 8 {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for range 1000 {
				inc()
			}
		}()
	}
	wg.Wait()
}

func TestUnsafeCounter(t *testing.T) {
	var c Counter
	hammer(c.Inc)
	t.Logf("final count: %d (without -race this test can still pass or fail by luck)", c.n)
}

func TestSafeCounter(t *testing.T) {
	var c SafeCounter
	hammer(c.Inc)
	if c.n != 8000 {
		t.Fatalf("got %d", c.n)
	}
}

Eight goroutines each increment a counter a thousand times. The unsafe Counter does c.n++, which is a read, an add and a write, with nothing stopping another goroutine from reading the old value in between. SafeCounter takes a mutex around the same statement.

echo "# without -race"
go test -v -count=1 -run Unsafe . 2>&1 | grep 'final count' | sed -E 's/^ +[a-z_]+\.go:[0-9]+: /  /'
echo
echo "# with -race, first lines of the report"
go test -race -count=1 -run Unsafe . 2>&1 | grep -vE '^\s*$' | head -14 | sed -E 's#/[^ ]*/(counter[a-z_]*\.go)#\1#; s/0x[0-9a-f]+/0xADDR/g; s/[0-9]+\.[0-9]+s/<time>/; s/goroutine [0-9]+/goroutine N/g'
echo
echo "# with -race, safe version"
go test -race -count=1 -run Safe . 2>&1 | sed -E 's/\t[0-9.]+s//'
echo
echo "# GORACE=halt_on_error=1"
GORACE=halt_on_error=1 go test -race -count=1 -run Unsafe . 2>&1 | grep -E 'WARNING|^FAIL\s*$|exit status' | sort -u | head -4

# without -race
  final count: 5810 (without -race this test can still pass or fail by luck)

# with -race, first lines of the report
==================
WARNING: DATA RACE
Read at 0xADDR by goroutine N:
  example.test/lab.(*Counter).Inc()
      counter.go:8 +0xADDR
  example.test/lab.(*Counter).Inc-fm()
      <autogenerated>:1 +0xADDR
  example.test/lab.hammer.func1()
      counter_test.go:15 +0xADDR
Previous write at 0xADDR by goroutine N:
  example.test/lab.(*Counter).Inc()
      counter.go:8 +0xADDR
  example.test/lab.(*Counter).Inc-fm()
      <autogenerated>:1 +0xADDR

# with -race, safe version
ok  	example.test/lab

# GORACE=halt_on_error=1
FAIL
WARNING: DATA RACE

The first block is the unsafe test without -race: the log shows final count: 5376. The goroutines did 8000 increments and 2624 were lost to overlapping updates. The test only logs the number, so it passes, but a test that asserted 8000 would fail unpredictably, and a test that asserted nothing about the count would never tell you. The lost updates are the symptom, and they depend on the machine: on a single core they might not occur at all.

Reading a report

With -race, the toolchain compiles the code with instrumentation that records every memory access and the synchronisation events around it. When two accesses to one address overlap without a happens-before relationship, the runtime prints a report and the test fails.

The report has a fixed structure. It starts with WARNING: DATA RACE, then describes the access that triggered it: Read at 0x... by goroutine N, followed by a stack. Then Previous write at 0x... by goroutine M with its own stack. Here both stacks have counter.go:8 at the top, which is c.n++: a read racing with an earlier write of the same variable. Later in the full report there are lines saying where each goroutine was created (Goroutine N (running) created at), which is how you find the go func() that started them. The recorded output is cut after the first stacks, with addresses replaced by 0xADDR, because a full report repeats for every pair of accesses.

Read it like this: find the two source lines at the top of each stack. They name the variable and both ways it is touched. Decide what should order them, a mutex, a channel or an atomic operation, and add it. Then check the creation stacks if you cannot see why two goroutines share the variable. The fixed version, SafeCounter, ran clean in the third block, printing only ok.

A report does not prove the line it points to is the whole bug. If a map is read in one goroutine and written in another, the stack names the lines, and the fix may be a lock somewhere else. Treat it as a pointer to a pair of accesses that need an ordering.

What the detector does not do

It is a dynamic tool, which sets its limits.

• It reports races that happen. If no test drives two goroutines through the code at once, the detector sees nothing. A clean run means no race was observed on the paths exercised. Write concurrent tests, like hammer above, for the code you share, and remember that t.Parallel tests share package state, as the parallel tests article explains.
• It has a limit on tracked goroutines and memory. A test that creates more than about 8,000 live goroutines at once can hit "race: limit on 8128 simultaneously alive goroutines is exceeded, dying". Reduce the number in race builds.
• It does not find deadlocks, leaks or logic races. Two goroutines taking locks in opposite order, or a check-then-act sequence that is atomic per step but not as a whole, are not data races in the detector's sense.
• It costs. Expect memory use up by five to ten times and execution time up by two to twenty times, depending on how much of the test is memory access. A suite that runs in a minute may take several.
• It needs cgo. -race requires CGO_ENABLED=1 and a C compiler on most platforms. A minimal Alpine or scratch based CI image needs gcc and musl-dev, or use a Debian based Go image for the race job.

Running it in CI

The usual shape is to run the whole suite with -race on every pull request. If that is too slow, there are two compromises, and one that does not work.

1. Race on the packages that use concurrency. Keep a list, or use go list with a filter, and run -race for those and the plain suite for the rest. The list rots, so review it when packages change.
2. Plain tests on pull requests, race on the main branch and nightly. A race merges, and is caught by the next run. This is the weakest option that still works.
3. Do not run it only before release. By then it finds twenty races, with nobody who remembers the code.

Add -count=1 so the test cache does not skip the run. Add -shuffle=on in a separate run to catch tests that depend on order. And set GORACE for the job. GORACE=halt_on_error=1 stops the process at the first report, which the last block of the lab shows: output reduced to WARNING: DATA RACE and FAIL. Without it, the detector keeps going and prints a report for every conflicting pair, which can make a log megabytes long. Other useful settings are log_path=/tmp/race to write reports to files that CI collects as artefacts, history_size to keep a longer access history so that "failed to restore the stack" disappears, and atexit_sleep_ms to adjust the delay at exit.

The tests that run under -race should also run with a time limit. Go's -timeout default of ten minutes is generous; set one that matches the suite, so a deadlock that the detector slowed down fails the job in a reasonable time. For the other side of that coin, hung tests are the subject of timed out test panics.

Fixing the common causes

• A shared variable written in one goroutine and read in another: a mutex, or send the value over a channel so the receive orders it.
• A loop variable captured by a goroutine: fixed by the per-iteration semantics since Go 1.22, but still seen in code on older language versions.
• A map accessed from several goroutines: sync.Mutex, sync.RWMutex, or sync.Map for the specific read-mostly patterns it suits.
• A counter or flag: sync/atomic types such as atomic.Int64 and atomic.Bool.
• A test and the code under test sharing a field after the test finishes: wait for goroutines with a WaitGroup or a context before the test returns.

When the detector's message is the thing you searched for, the dedicated article on the data race warning in go test goes through the fixes one at a time. The Go team's race detector article documents the report format and the GORACE options, and the memory model it checks against is in the Go memory model.