Safe Counters in Go With Atomics Mutexes and Sharding

A shared counter is the smallest concurrent data structure, and the one most programs have: requests served, bytes written, cache hits. It is also a good case study, because the obvious fixes, a mutex and then an atomic, behave very differently as you add cores, and the fix after that only matters if you understand why. This article benchmarks four implementations at one, four and ten cores and explains the result with one idea: contention on a single cache line.

It builds on atomics, mutexes and the earlier comparison in channels or mutexes, and it feeds into lock contention. It belongs to the concurrency patterns cluster. All numbers are from an Apple M1 Max on Go 1.25.6, so read them as shapes.

The four counters

The first two are what you would write first. The second pair is the interesting one. A sharded counter keeps an array of independent cells, and each increment goes to one of them; reading sums every cell. Spreading the writes means goroutines on different cores usually touch different memory. The padded version adds 56 bytes after each 8-byte cell so that every cell sits on its own 64-byte cache line, and the unpadded version packs the cells together:

package main

import (
	"math/rand/v2"
	"sync"
	"sync/atomic"
	"testing"
)

type paddedCounter struct {
	cells [64]struct {
		n atomic.Int64
		_ [56]byte // keep each cell on its own 64-byte cache line
	}
}

func (c *paddedCounter) Add() { c.cells[rand.Uint32()%64].n.Add(1) }

func (c *paddedCounter) Load() int64 {
	var total int64
	for i := range c.cells {
		total += c.cells[i].n.Load()
	}
	return total
}

type packedCounter struct {
	cells [64]atomic.Int64 // eight cells share each cache line
}

func (c *packedCounter) Add() { c.cells[rand.Uint32()%64].Add(1) }

func BenchmarkMutex(b *testing.B) {
	var mu sync.Mutex
	n := 0
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			mu.Lock()
			n++
			mu.Unlock()
		}
	})
}

func BenchmarkAtomic(b *testing.B) {
	var n atomic.Int64
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			n.Add(1)
		}
	})
}

func BenchmarkShardedPadded(b *testing.B) {
	var c paddedCounter
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			c.Add()
		}
	})
}

func BenchmarkShardedPacked(b *testing.B) {
	var c packedCounter
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			c.Add()
		}
	})
}

goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
BenchmarkMutex               	85879166	        13.73 ns/op	       0 B/op	       0 allocs/op
BenchmarkMutex-4             	14104288	        76.24 ns/op	       0 B/op	       0 allocs/op
BenchmarkMutex-10            	10305326	       113.7 ns/op	       0 B/op	       0 allocs/op
BenchmarkAtomic              	169764355	         7.022 ns/op	       0 B/op	       0 allocs/op
BenchmarkAtomic-4            	42195144	        33.03 ns/op	       0 B/op	       0 allocs/op
BenchmarkAtomic-10           	21518085	        56.72 ns/op	       0 B/op	       0 allocs/op
BenchmarkShardedPadded       	170617184	         7.285 ns/op	       0 B/op	       0 allocs/op
BenchmarkShardedPadded-4     	100000000	        11.49 ns/op	       0 B/op	       0 allocs/op
BenchmarkShardedPadded-10    	90597086	        12.51 ns/op	       0 B/op	       0 allocs/op
BenchmarkShardedPacked       	173197525	         7.074 ns/op	       0 B/op	       0 allocs/op
BenchmarkShardedPacked-4     	47849193	        22.37 ns/op	       0 B/op	       0 allocs/op
BenchmarkShardedPacked-10    	41928597	        27.36 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	example.test/lab	17.684s

b.RunParallel runs the loop on as many goroutines as GOMAXPROCS, and -cpu 1,4,10 repeats every benchmark at each setting. The reported ns/op is wall time divided by total operations across all goroutines, so a number that grows with cores means the operations are getting slower in aggregate, not that any single one is. The shard is picked with rand.Uint32() from math/rand/v2, whose top-level functions use a per-thread generator that does not take a lock, so choosing a shard does not itself become the contended resource.

The numbers

On a single core the atomic is twice as fast as the mutex, and all the sharded variants equal the atomic, since there is no contention to spread. As cores are added, the single atomic stops being cheap: at ten cores it costs eight times what it did at one. The mutex follows the same curve at roughly double the level. The padded sharded counter barely moves: 7 ns becomes 13 ns across a tenfold increase in cores.

Why an atomic gets slower

An atomic add is a single instruction, so it is natural to expect it to scale. It does not, because of how cores share memory. Each core has its own cache, and a cache line (64 bytes on this hardware) can be owned for writing by one core at a time. To increment the counter, a core must first take exclusive ownership of the line, which invalidates the copy in every other core's cache. With ten cores all incrementing the same line, the line spends its life moving between caches, and each increment waits for that transfer. The instruction is fast; the coordination is not. This is called cache line contention, and it is the real cost under both the mutex (whose state lives on a line of its own, in addition to the counter) and the atomic.

That is also why the packed shards helped only partly. Eight 8-byte cells fit in one 64-byte line, so goroutines picking different cells still contend when those cells share a line. This effect is called false sharing: no two goroutines use the same variable, but they use the same line. Padding each cell to a full line removes it, and the cost per increment dropped from about 27 ns to about 13 ns at ten cores, a change that would be invisible in code review and is worth more than doubling throughput.

What sharding costs you

Sharding trades write speed for read cost and consistency. A Load visits 64 cells instead of one, so reading is dozens of times slower than a single atomic load. And the total is not a snapshot: while writers run, the sum reads each cell at a slightly different moment, so it can be off by whatever was added during the scan. For a metric you export every few seconds, neither matters. For anything where an exact value drives a decision, such as a rate limit, a quota or a reference count that frees memory at zero, a sharded counter is the wrong tool; use a single atomic or a mutex and accept the contention.

Memory is the third price. Sixty-four padded cells take 4 KiB per counter, against 8 bytes for an atomic. That is nothing for one global request counter and a lot for a counter per user. Size the shard count to roughly the number of cores (a power of two so the modulo is cheap), not to a large fixed number, and make it a field of the type so tests can use a small one.

Choosing, in practice

The decision rule I use:

1. Start with atomic.Int64. It is correct, readable and as fast as anything until many cores hammer it. Most counters never leave this step.
2. Use a mutex when several values must change together, such as a count and a sum. An atomic per field cannot give a consistent pair.
3. Shard only after a profile shows the counter itself, an atomic add high in a CPU profile or contention on its line, is limiting a hot path. Then pad.
4. Count privately and merge when the loop is yours. If a worker increments a local int and adds it to the shared counter once at the end, or every thousand iterations, the shared cost vanishes. The partitioning example does this.

Counting per goroutine and merging is usually the best answer and needs no clever type at all. The sharded counter is for the case where you cannot control the callers, such as a library counting events from arbitrary request goroutines.

A reminder about what correctness needs

None of this is about correctness; all four counters are race-free. The unsafe version, n++ on a shared integer, is the one in data races explained, where eight goroutines lost most of their increments. If you need a counter in a hurry, the atomic is the answer, and atomic.Int64 has been available as a type since Go 1.19, which removes the old atomic.AddInt64(&n, 1) plumbing and the alignment pitfalls that came with it on 32-bit platforms.

The standard library takes the sharding idea seriously in places. sync.Pool keeps per-processor local pools to avoid a shared lock, and the runtime's own statistics counters are per-P for the same reason. The sync/atomic documentation describes the typed values, and the Go FAQ and race detector notes on go.dev cover what counts as a safe access. For a deeper look at why a lock slows down, the next article in the series is lock contention.