Pick the Right Worker Pool Size for Your Go Workload

"How many workers should I use?" is the question every pool author eventually asks, and the answers on the internet disagree: the number of cores, twice the number of cores, a hundred, as many as you can. They disagree because they are answering different workloads. A pool that computes hashes and a pool that calls an HTTP API have different bottlenecks, and each has a different right size.

This article measures both kinds across a range of sizes on one machine and reads the curves. It builds on worker pools and GOMAXPROCS in containers. The machine is an Apple M1 Max with ten cores, so the interesting number for the CPU case is ten. The results show the shape; your hardware will move the numbers.

The two workloads

A CPU-bound job is one where the worker spends its time computing: hashing, compressing, parsing, rendering. The limit is the processor. Adding workers beyond the number of cores cannot make the machine compute faster; there is nothing for the extra workers to run on.

An I/O-bound job spends its time waiting: for a database, an HTTP call, a disk, a message broker. The limit is not the processor, because while one worker waits, the processor is free to run another. Here more workers keep helping until something else saturates.

The benchmark below uses a pool of N goroutines draining a channel, once with a job that burns CPU (a loop of arithmetic) and once with a job that sleeps for a millisecond (standing in for a network call, with latency and no CPU). Each iteration is one job, so the nanoseconds per operation are the inverse of throughput: lower is better.

package main

import (
	"fmt"
	"sync"
	"sync/atomic"
	"testing"
	"time"
)

var sink atomic.Int64

func burn(n int) int {
	x := 0
	for i := 0; i < n; i++ {
		x += i * i
	}
	return x
}

func cpuJob() int     { return burn(20_000) }    // a few microseconds
func cpuJobLong() int { return burn(2_000_000) } // a few hundred microseconds

func ioJob() int {
	time.Sleep(time.Millisecond)
	return 0
}

func runPool(b *testing.B, workers int, job func() int) {
	jobs := make(chan struct{}, workers)
	var wg sync.WaitGroup
	for range workers {
		wg.Go(func() {
			acc := 0 // each worker keeps its own total, so workers do not share a cache line
			for range jobs {
				acc += job()
			}
			sink.Add(int64(acc))
		})
	}
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		jobs <- struct{}{}
	}
	close(jobs)
	wg.Wait()
}

func BenchmarkCPUBound(b *testing.B) {
	for _, n := range []int{1, 2, 5, 10, 20, 100} {
		b.Run(fmt.Sprintf("workers=%d", n), func(b *testing.B) { runPool(b, n, cpuJob) })
	}
}

func BenchmarkCPUBoundLongJobs(b *testing.B) {
	for _, n := range []int{1, 5, 10, 20, 100} {
		b.Run(fmt.Sprintf("workers=%d", n), func(b *testing.B) { runPool(b, n, cpuJobLong) })
	}
}

func BenchmarkIOBound(b *testing.B) {
	for _, n := range []int{1, 10, 50, 100, 500, 2000} {
		b.Run(fmt.Sprintf("workers=%d", n), func(b *testing.B) { runPool(b, n, ioJob) })
	}
}

goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
BenchmarkCPUBound/workers=1-10  	   44290	      7066 ns/op
BenchmarkCPUBound/workers=1-10  	   51522	      7088 ns/op
BenchmarkCPUBound/workers=1-10  	   51108	      7004 ns/op
BenchmarkCPUBound/workers=2-10  	   97150	      3793 ns/op
BenchmarkCPUBound/workers=2-10  	   94135	      3729 ns/op
BenchmarkCPUBound/workers=2-10  	   95418	      3728 ns/op
BenchmarkCPUBound/workers=5-10  	  157808	      2331 ns/op
BenchmarkCPUBound/workers=5-10  	  153393	      2414 ns/op
BenchmarkCPUBound/workers=5-10  	  151728	      2314 ns/op
BenchmarkCPUBound/workers=10-10 	  190303	      1978 ns/op
BenchmarkCPUBound/workers=10-10 	  184726	      1970 ns/op
BenchmarkCPUBound/workers=10-10 	  188626	      1904 ns/op
BenchmarkCPUBound/workers=20-10 	  230413	      1628 ns/op
BenchmarkCPUBound/workers=20-10 	  227049	      1625 ns/op
BenchmarkCPUBound/workers=20-10 	  222565	      1626 ns/op
BenchmarkCPUBound/workers=100-10         	  269986	      1371 ns/op
BenchmarkCPUBound/workers=100-10         	  267592	      1345 ns/op
BenchmarkCPUBound/workers=100-10         	  269529	      1401 ns/op
BenchmarkCPUBoundLongJobs/workers=1-10   	     550	    641660 ns/op
BenchmarkCPUBoundLongJobs/workers=1-10   	     567	    638801 ns/op
BenchmarkCPUBoundLongJobs/workers=1-10   	     571	    651016 ns/op
BenchmarkCPUBoundLongJobs/workers=5-10   	    2721	    131417 ns/op
BenchmarkCPUBoundLongJobs/workers=5-10   	    2713	    132431 ns/op
BenchmarkCPUBoundLongJobs/workers=5-10   	    2692	    132620 ns/op
BenchmarkCPUBoundLongJobs/workers=10-10  	    4542	     78522 ns/op
BenchmarkCPUBoundLongJobs/workers=10-10  	    3891	     79857 ns/op
BenchmarkCPUBoundLongJobs/workers=10-10  	    4532	     76753 ns/op
BenchmarkCPUBoundLongJobs/workers=20-10  	    4419	     77144 ns/op
BenchmarkCPUBoundLongJobs/workers=20-10  	    4546	     83759 ns/op
BenchmarkCPUBoundLongJobs/workers=20-10  	    4106	     79353 ns/op
BenchmarkCPUBoundLongJobs/workers=100-10 	    4498	     79058 ns/op
BenchmarkCPUBoundLongJobs/workers=100-10 	    4189	     79768 ns/op
BenchmarkCPUBoundLongJobs/workers=100-10 	    4500	     77464 ns/op
BenchmarkIOBound/workers=1-10            	     289	   1289045 ns/op
BenchmarkIOBound/workers=1-10            	     279	   1360653 ns/op
BenchmarkIOBound/workers=1-10            	     282	   1253530 ns/op
BenchmarkIOBound/workers=10-10           	    2876	    128003 ns/op
BenchmarkIOBound/workers=10-10           	    2900	    131211 ns/op
BenchmarkIOBound/workers=10-10           	    2839	    126034 ns/op
BenchmarkIOBound/workers=50-10           	   14250	     26788 ns/op
BenchmarkIOBound/workers=50-10           	   13305	     25926 ns/op
BenchmarkIOBound/workers=50-10           	   14116	     26961 ns/op
BenchmarkIOBound/workers=100-10          	   28699	     13182 ns/op
BenchmarkIOBound/workers=100-10          	   28075	     12580 ns/op
BenchmarkIOBound/workers=100-10          	   28074	     13005 ns/op
BenchmarkIOBound/workers=500-10          	  146392	      2399 ns/op
BenchmarkIOBound/workers=500-10          	  148480	      2362 ns/op
BenchmarkIOBound/workers=500-10          	  152035	      2399 ns/op
BenchmarkIOBound/workers=2000-10         	  687880	       526.8 ns/op
BenchmarkIOBound/workers=2000-10         	  565844	       548.4 ns/op
BenchmarkIOBound/workers=2000-10         	  563527	       542.3 ns/op
PASS
ok  	example.test/lab	21.402s

CPU-bound results

Medians of three runs. The first column is the pool size; the others are microseconds per job, where lower is better, and the speedup compared with one worker.

The long-job column is the clean result. Throughput scales almost linearly up to the core count (4.8 times faster at 5 workers, 8.2 times at 10) and then stops: 20 and 100 workers are no faster than 10, and marginally slower. That is the textbook CPU-bound curve, and the plateau sits at the number of cores. It is 8.2 times and not 10 because the M1 Max's ten cores are eight performance cores and two efficiency cores, and because clock speed drops when more cores are busy.

The short-job column behaves differently, and the difference is the lesson. With jobs of about seven microseconds, the pool still gains up to 10 workers (3.6 times), but it keeps improving beyond the core count: 20 workers are 4.3 times faster than one, and 100 workers 5.2 times. More workers than cores helped here, which a rule of "workers equal cores" would have missed. I did not isolate the cause with a profile, but the long-job column points to a reading: when a job takes only microseconds, a large part of each job's cost is the channel handoff, with the worker parked waiting for the producer to wake it. A worker that is parked leaves its core idle, and a surplus of workers means a runnable goroutine is nearly always available to fill that gap. When jobs take hundreds of microseconds, the handoff cost vanishes into the work and the extra workers add nothing, exactly as the long-job column shows.

I/O-bound results

Each job sleeps for a millisecond, so one worker can complete a bit under a thousand jobs a second.

Throughput is almost exactly proportional to the worker count, all the way to 2,000 workers on a ten-core machine, because a sleeping goroutine uses no CPU. The curve only bends when something else becomes the limit, and in this synthetic job nothing does. The number to notice is the last: 2,000 workers each completing about 920 jobs a second is 1.84 million jobs a second, 92% of the theoretical ceiling of workers divided by latency (2,000 / 0.001 s = 2 million). The gap is time.Sleep(time.Millisecond) taking a little longer than a millisecond in practice (the single worker's 1.289 ms per job shows it). A real dependency will not scale like this. Your database allows N connections, your downstream API rate-limits, your network has a bandwidth, and the I/O-bound curve flattens at whichever of them is smallest.

Reading the curves

CPU-bound work. Start with runtime.GOMAXPROCS(0) workers (GOMAXPROCS in containers). That is where throughput stops rising for substantial jobs. If your jobs are tiny, consider batching them (a worker takes a slice of items per receive) instead of adding workers; that attacks the handoff cost directly and lets the number of workers stay at the core count.

I/O-bound work. The right size has nothing to do with cores. It is determined by what the work waits on. If each job is one call to a service that can serve 50 concurrent requests, a pool of 50 saturates it, and a pool of 500 only builds a queue inside the service, with higher latency and more timeouts for everyone. Find the dependency's concurrency limit (its connection pool, its documented rate, its observed knee in the latency curve) and size the pool at or just below it.

Mixed work. Jobs that compute and then wait, or wait and then compute, sit between the two. Size for the waiting part, then check that the computing part does not saturate the cores: CPU utilisation near 100% at your pool size means you have gone past the point of benefit.

Sizing from first principles: Little's law

A starting number for the I/O case comes from Little's law, a result from queueing theory: the average number of items in a system equals the arrival rate times the average time each spends in it. For a pool: concurrency needed = throughput wanted x latency per job. To sustain 2,000 jobs a second when each takes 50 ms, you need 2,000 x 0.05 = 100 jobs in flight, so 100 workers, plus headroom for variation.

package main

import (
	"fmt"
	"math"
	"time"
)

// poolSize returns the workers needed to sustain rate jobs per second at the
// given latency, with headroom, capped by what the dependency can take.
func poolSize(rate float64, latency time.Duration, headroom float64, dependencyLimit int) int {
	need := rate * latency.Seconds() * headroom
	n := int(math.Ceil(need))
	if dependencyLimit > 0 && n > dependencyLimit {
		return dependencyLimit
	}
	return n
}

func main() {
	fmt.Println("2000/s at 50ms, 1.5x headroom:        ", poolSize(2000, 50*time.Millisecond, 1.5, 0))
	fmt.Println("2000/s at 50ms, but the DB takes 64:  ", poolSize(2000, 50*time.Millisecond, 1.5, 64))
	fmt.Println("300/s at 200ms, 1.25x headroom:       ", poolSize(300, 200*time.Millisecond, 1.25, 0))
}

2000/s at 50ms, 1.5x headroom:         150
2000/s at 50ms, but the DB takes 64:   64
300/s at 200ms, 1.25x headroom:        75

The function multiplies rate by latency and adds headroom, then caps the answer at what the downstream can take. The second line is the important one: the formula asks for 150 workers, the database accepts 64, so the pool is 64 and the target rate is out of reach. When the arithmetic says you need more concurrency than the dependency allows, you have found your real bottleneck before writing any code, and the options are to raise the dependency's limit, add caching so fewer jobs wait on it, or accept a lower rate. Little's law uses averages, so check it against measured latency at the percentile you care about (p95 latency gives a safer sizing than the mean) and re-measure after changes.

A procedure

1. Classify the job: CPU-bound, I/O-bound or mixed. A profile settles it.
2. Start from the formula: cores for CPU work, rate x latency for I/O work, capped by the dependency.
3. Run a benchmark or a load test with sizes below and above the guess, on the production hardware or a container with the same CPU limit.
4. Look at throughput, latency percentiles, CPU utilisation and the dependency's error rate. The best size is the smallest one that reaches the plateau.
5. Make the size configurable, and expose queue depth as a metric so you can see when it is wrong.

A pool that is too small shows as a growing queue and idle dependencies. A pool that is too large shows as high latency, timeouts and errors in the dependency, and memory. Prefer the smaller of two sizes that perform equally: it has more margin before it harms the thing it calls. For the pool itself, see building a worker pool, and for the bound without a queue, semaphores. Little's law is described in Wikipedia's article on the result, and the Go benchmark tooling is documented in the testing package.