Lock Contention in Go and How to Spot It in Production

A Go service is slow, the CPU graph shows 30%, and nothing in the logs is wrong. Adding replicas helps a little, adding cores to each replica does nothing. The usual suspect is a lock that every request passes through, held a little too long by whoever has it. Goroutines arrive, find the mutex taken, park, and wait their turn, and while they wait the processor sits idle. This is lock contention, and it is the commonest reason a concurrent Go program does not scale.

This guide covers how to recognise it, how to measure it with tools that are safe to leave on in production, how to find the guilty lock in the mutex profile, and how to fix it in the order that usually works. It is part of the concurrency patterns cluster and builds on mutexes, the counter benchmark and sharded maps. The two profile types get a closer look in the mutex profile and block profile. Everything ran on Go 1.25.6.

What contention looks like

Contention has a recognisable shape on a dashboard. Throughput rises with load up to some point and then flattens, even though the machine is not busy. Latency percentiles climb while the median stays calm, because the unlucky requests are the ones that queue longest. CPU utilization is lower than the request rate would suggest. If you double the number of workers the throughput stays put and p99 gets worse, since more goroutines now compete for the same lock. A CPU profile of such a service looks unremarkable, because waiting is not CPU time: the profile shows the work that gets done, not the queue it waited in.

That last point is why contention hides. The tools that most people reach for first, top and the CPU profile, are blind to it. You need a measurement of time spent waiting.

A reproduction

This program has the textbook mistake: a cache whose get holds its mutex while doing slow work, eight goroutines hammering it. It runs the contended version, then a fixed one that does the slow work outside the lock, and reports how long each took and how much time goroutines spent blocked on mutexes in total, from the runtime's own counter:

package main

import (
	"fmt"
	"runtime/metrics"
	"sync"
	"time"
)

type cache struct {
	mu sync.Mutex
	m  map[int]int
}

// slowGet does its slow work while holding the lock.
func (c *cache) slowGet(k int) int {
	c.mu.Lock()
	defer c.mu.Unlock()
	time.Sleep(50 * time.Microsecond) // stands in for parsing, formatting, a syscall
	return c.m[k]
}

// fastGet holds the lock only for the map lookup.
func (c *cache) fastGet(k int) int {
	c.mu.Lock()
	v := c.m[k]
	c.mu.Unlock()
	time.Sleep(50 * time.Microsecond) // the same work, now outside the critical section
	return v
}

func mutexWait() float64 {
	s := []metrics.Sample{{Name: "/sync/mutex/wait/total:seconds"}}
	metrics.Read(s)
	return s[0].Value.Float64()
}

func measure(name string, get func(int) int) {
	before := mutexWait()
	start := time.Now()
	var wg sync.WaitGroup
	for range 8 {
		wg.Go(func() {
			for range 200 {
				get(1)
			}
		})
	}
	wg.Wait()
	fmt.Printf("%-8s elapsed %-6v goroutine time blocked on mutexes %.3fs\n",
		name, time.Since(start).Round(time.Millisecond), mutexWait()-before)
}

func main() {
	c := &cache{m: map[int]int{1: 1}}
	measure("slow", c.slowGet)
	measure("fast", c.fastGet)
}

slow     elapsed 113ms  goroutine time blocked on mutexes 0.494s
fast     elapsed 14ms   goroutine time blocked on mutexes 0.000s

The slow version serialized 1,600 operations of 50 microseconds each, so about 113 ms in total, and during that time the eight goroutines together spent roughly half a second blocked on the mutex: seven of them were always waiting while one worked. The fast version does the same work but outside the lock, so the eight goroutines overlap their sleeps. The elapsed time falls to 14 ms and the blocked time is close to zero. The counter I read is the interesting part. /sync/mutex/wait/total:seconds is a cumulative metric exposed by the runtime/metrics package: the total time goroutines have spent blocked on sync.Mutex and sync.RWMutex. It costs almost nothing to read and is on by default, unlike profiling.

Make it a metric

Because the counter is cheap, export it. Sample it every few seconds and publish the rate of increase. The value is in goroutine-seconds, so a rate of 3.0 means that, on average, three goroutines were blocked on a mutex at any moment. Compare it with your goroutine count and your core count: a service with 40 busy goroutines and a mutex-wait rate of 25 is mostly queueing. If you use the Prometheus client library, its Go collector already exposes this metric as go_sync_mutex_wait_total_seconds when runtime metrics are enabled, so a rate query is one line.

The alarm I would set is not an absolute threshold but a change: the rate jumping after a deploy is a regression in locking, and the diff is usually one function that grew work inside a critical section.

Find the lock: the mutex profile

The metric says that contention exists. The mutex profile says where. It is disabled by default because sampling has a cost, and runtime.SetMutexProfileFraction(n) turns it on: on average one in every n contention events is recorded, and a value of 1 records all of them. In a test or a reproduction, use 1. In production, 100 is a common setting. This program turns it on, runs the contended workload and prints the head of the profile in its text format:

package main

import (
	"os"
	"runtime"
	"runtime/pprof"
	"sync"
	"time"
)

type cache struct {
	mu sync.Mutex
	m  map[int]int
}

func (c *cache) get(k int) int {
	c.mu.Lock()
	defer c.mu.Unlock()
	time.Sleep(50 * time.Microsecond)
	return c.m[k]
}

func main() {
	runtime.SetMutexProfileFraction(1)
	c := &cache{m: map[int]int{1: 1}}
	var wg sync.WaitGroup
	for range 8 {
		wg.Go(func() {
			for range 200 {
				c.get(1)
			}
		})
	}
	wg.Wait()
	pprof.Lookup("mutex").WriteTo(os.Stdout, 1)
}

--- mutex:
cycles/second=999999643
sampling period=1
500907759 1593 @ 0x100d26e05 0x100d26da4 0x100d27004 0x100cdc16c 0x100cd6d64
#	0x100d26e04	sync.(*Mutex).Unlock+0x24	sync/mutex.go:65
#	0x100d26da3	main.(*cache).get+0xd3		example.test/lab/main.go:20
#	0x100d27003	main.main.func1+0x33		example.test/lab/main.go:30
#	0x100cdc16b	sync.(*WaitGroup).Go.func1+0x4b	sync/waitgroup.go:239

The profile has one record: the first number is the total delay in cycles, the second is the number of contention events, and the stack below it is where the contended mutex was unlocked. That detail trips people up. The profile does not point at the goroutines that waited; it points at the code that held the lock while they waited, which is the code you need to change. Here the top frame is sync.(*Mutex).Unlock and the next one is main.(*cache).get: the delay was attributed to the unlock inside get, so get is the function whose critical section is too long. In a real service you would open the profile with go tool pprof instead of reading the text; the mechanics of that are in the next article, along with the block profile, which covers waits on channels and other primitives as well.

Fix it, in this order

The order matters because each step is cheaper and less invasive than the next.

1. Shorten the critical section. Look at what the code does between Lock and Unlock. Anything that does not touch the protected data does not belong there: network calls, disk I/O, logging, formatting, hashing, sleeping, calling user-supplied callbacks. Copy the data you need under the lock, release it, do the work. The fastGet above is the whole idea. This fixes more contention problems than every other technique combined, because the most common cause is a lock held across something slow. A defer mu.Unlock() at the top of a long function is the usual culprit, since it makes the whole function the critical section.

2. Split the lock. One mutex guarding unrelated fields makes unrelated operations wait for each other. If a struct has a counters section and a configuration section, two mutexes let those proceed independently. Keep each lock next to the data it guards, and document the field list it covers.

3. Use the right kind of lock. If most operations only read, a sync.RWMutex lets readers run in parallel (RWMutex). The caveat is that an RWMutex costs more per operation than a Mutex, so it pays only for read-heavy workloads with a non-trivial critical section; measure before converting.

4. Replace the lock with an atomic. A single counter, flag or pointer does not need a mutex (atomics). For a pointer to immutable data, atomic.Pointer lets readers proceed with no lock at all.

5. Shard. Split one hot structure into independent parts selected by key, so goroutines on different keys take different locks (sharded maps, sharded counters). This is more code and more memory and it complicates operations that span the whole structure, so it comes after the cheaper steps.

6. Remove the sharing. Per-goroutine state merged at the end, an owner goroutine that serializes access (the actor pattern), or immutable data copied on update (copy on write) avoid the lock entirely. Sometimes this is a redesign, and sometimes it is the right answer.

How much serial time is too much

Contention is an instance of Amdahl's law, and the arithmetic is worth doing once. If a fraction s of each request runs inside one lock, no number of cores can push throughput past 1/s times the single-core rate. A request that spends 100 microseconds in a critical section tops out at 10,000 operations per second per lock, whatever the hardware. At 20 microseconds the ceiling is 50,000. So the useful question about a lock is not "is it contended right now?" but "how long is it held, and how many operations per second do I need through it?". Multiply the two: if the product approaches 1, the lock is a saturated resource and every additional request queues. Hold time of 50 microseconds with a demand of 15,000 operations a second is a product of 0.75, which is why a modest traffic increase can turn a calm service into a queue overnight.

This also explains why contention appears suddenly. Below saturation, waits are short and rare. Near it, queueing delay grows steeply, the same shape as a road at rush hour, and a small change in load or in hold time makes a large change in latency. A service that has been fine for months can fall off that cliff with a new feature that adds ten microseconds under a lock.

The usual culprits

When I read a mutex profile, the lock is held across one of a few things almost every time.

Logging. log.Printf inside a critical section writes to a file or a pipe, and a slow disk or a blocked pipe turns every waiter into a bystander. Log after Unlock, using values copied while locked.

A call to something you do not control. An interface method, a callback, a Stringer. You cannot know how long it takes, or whether it takes the same lock again. Holding a lock across such a call is a deadlock risk as well as a latency one.

Allocation and formatting. Building a large response, marshalling JSON, formatting a string. These are CPU work that doesn't need exclusive access to the data; compute them from a copy.

A lazy load inside a getter. The classic get that checks the map, misses, fetches from the database, and stores the result, all under one lock. Every other caller waits for the database. The fix is either to release the lock around the fetch and handle the duplicate-fetch race, or to use singleflight so that concurrent misses for the same key share one fetch while hits are not blocked.

func (c *Cache) Get(key string) (string, error) {
	c.mu.RLock()
	v, ok := c.m[key]
	c.mu.RUnlock()
	if ok {
		return v, nil
	}
	v, err, _ := c.group.Do(key, func() (any, error) { // one fetch per key at a time
		return c.load(key) // runs with no lock held
	})
	if err != nil {
		return "", err
	}
	c.mu.Lock()
	c.m[key] = v.(string)
	c.mu.Unlock()
	return v.(string), nil
}

Every lock in that function covers a map access and nothing else, and the slow database call runs with no lock held. That is the shape the previous section's first rule produces, and it is the single most valuable review habit for concurrent Go: read each Lock and ask what is between it and the matching Unlock.

Pitfalls when measuring

Microbenchmarks mislead about contention. A single-goroutine benchmark of a locked function shows the uncontended cost of the mutex, a few nanoseconds, and says nothing about how it behaves under a real load. Use b.RunParallel and run with -cpu 1,4,N so you see the curve, as in the counter article.

The profile is sampled. With a fraction of 100 you see roughly one event in a hundred, which is plenty for a hot lock and can miss a rare one. Raise the rate for a short window if you need detail, and lower it again.

Waiting is not always a mutex. A service blocked on a channel, a database connection pool, or a semaphore will not show up in the mutex metric. The block profile and the execution tracer see those (execution tracer). If the mutex wait rate is flat and the service is still idle, look there.

A fix that moves the problem. Shortening a critical section sometimes turns lock contention into contention for the thing the work depends on (a downstream service, a connection pool). That is progress, because you now see the real limit, but expect the next bottleneck to be a different one.

A short routine for production

When latency climbs and CPU does not, I check four things in order. First, the mutex-wait rate from the runtime metric: if it is near zero, contention on sync locks is not the problem. Second, if it is high, take a mutex profile for thirty seconds with a fraction of 100 and read the top stacks. Third, open the function the top stack names and look for work inside the critical section. Fourth, make the smallest change that removes it, deploy, and compare the metric's rate before and after. The metric turns a performance guess into an experiment with a number at the end.

The standard references are the runtime/metrics package documentation, which lists the mutex wait metric among the supported ones, the diagnostics guide on go.dev, which describes the mutex and block profiles in the context of the other tools, and the sync package documentation for the locks themselves.