Go Concurrency Patterns That Survive Production Traffic

Go makes concurrency cheap to start and easy to get wrong. A goroutine costs a few kilobytes and one keyword, so a program can have a hundred thousand of them, and every one is a small decision that nobody reviews: who stops it, who hears about its failure, and what happens when it is slower than the thing feeding it. The language gives you the pieces. It does not give you the discipline, and the discipline is what separates a service that runs for a year from one that needs a restart every Friday.

This is the hub for the concurrency cluster on this site. It is a map, written to be read straight through once and then used as an index. For each pattern it says what problem it solves, when it is the wrong tool, and where the detailed article lives. A runnable example near the end combines several patterns into one small function that I would be comfortable shipping. Everything here was run on Go 1.25.6 on an Apple M1 Max, and the linked articles carry their own measured output.

Start with the mental model

Concurrency in Go is about structure, not speed. Splitting work across goroutines can make a program faster on a multi-core machine, but the first reason to reach for it is that some problems are naturally several things happening at once: a server handling many connections, a program waiting on a network call while it parses a file, a pipeline whose stages run at different rates. Parallelism, running things at the same instant on different cores, is a consequence that the runtime arranges for you.

A goroutine is a function running on its own stack, scheduled by the Go runtime onto a small number of operating system threads. Goroutines explained covers what one is, how much it costs and why a program with a million of them can still work. The scheduler that places them on threads is the subject of the Go scheduler explained, and it matters more than most people expect, because the way goroutines are preempted, parked and woken explains behaviours that look like bugs, such as a tight loop starving other goroutines on older versions. A related trap shows up in containers: the runtime sizes itself from the number of CPUs it sees, and a container with a CPU limit can see far more than it may use. GOMAXPROCS in containers shows the effect with numbers.

Two ideas carry through every pattern below. The first is ownership: at any moment, one goroutine owns a piece of data and is allowed to change it, and handing it to another goroutine is an explicit act. The second is lifetime: every goroutine you start has a defined end, and some piece of code waits for that end. Almost every concurrency bug I have debugged in production was a violation of one of these two. A data race is two goroutines both believing they own a value. A leak is a goroutine whose lifetime nobody owns.

The Go memory model is the formal statement of what one goroutine is guaranteed to see of another's writes, and it is shorter than people fear. The practical summary is in the Go memory model: if two goroutines touch the same variable and at least one writes, they need synchronisation, and synchronisation means a channel operation, a mutex, a sync.WaitGroup or an atomic. Anything else, including a time.Sleep that "should be long enough", is a race with a good chance of passing every test you run.

Channels: moving data and signals between goroutines

A channel is a typed conduit between goroutines. Sending on it and receiving from it are synchronisation points, so a value sent is visible to the receiver, along with everything the sender did before sending. That is why channels are the right tool when you are handing something over: a job to a worker, a result to a collector, a signal that a phase is done.

Channels explained is the full guide. The decision you make most often is between an unbuffered and a buffered channel. An unbuffered channel makes sender and receiver meet, which gives you a handshake and natural backpressure. A buffered one lets the sender run ahead by a fixed amount, which smooths short bursts and hides a slow consumer until the buffer fills. Buffered versus unbuffered channels shows the difference in behaviour and in timing. My rule is to start unbuffered, and to add a buffer only for a reason I can state, such as "up to N results may arrive before the collector starts" or "the producer must not block on a slow log sink". A buffer that exists because the program deadlocked without one is a bug that has been moved somewhere harder to see.

Closing a channel is a signal from the sender that no more values are coming. Only the sender closes, a receiver never does, and closing twice is a panic. Closing a channel lays out the rules, and the panic when you send on a closed channel explains the error message, the usual causes and the three fixes. When several goroutines send on the same channel, no single one of them knows when it is safe to close, so a coordinator closes after a WaitGroup says every sender is finished. That small pattern appears in nearly every fan-in.

The other channel failure everybody meets is the deadlock. The runtime detects the case where every goroutine is blocked and the program can never proceed, and it ends the process with a fatal error. All goroutines are asleep explains how to read that report and why the detector only catches the total case: a program that deadlocks while a timer or a network listener is still alive just hangs, which is much worse and which the leak tools below are for.

Select: waiting on more than one thing

select waits on several channel operations and runs the one that is ready, picking at random when more than one is. It is the building block for timeouts, cancellation, non-blocking sends and multiplexing, and the select statement goes through each of those. Almost every blocking operation in a production service should be inside a select that also watches ctx.Done(), because a bare <-ch or ch <- v has no way out except the other side showing up.

One property of select is useful enough to deserve its own article: a nil channel blocks forever, so setting a channel variable to nil takes its case out of the select. Nil channels in select shows how that turns a merge loop that must drain two sources into a dozen lines instead of a state machine.

Shared state: when a lock is the right answer

Channels are not always the answer. The Go proverb says to share memory by communicating, and it is good advice for handing off work, but a counter, a cache or a set of connections is state that many goroutines read and update, and a mutex around it is shorter, faster and easier to read than a goroutine whose only job is to serialise access. I use both in the same program without guilt.

The primitives and where each fits:

A few judgements that are not obvious from the documentation. A WaitGroup is the right way to wait for goroutines whose results you collect another way, and since Go 1.25 its Go method removes the classic mistake of calling Add after the goroutine has already started. A plain Mutex is usually faster than an RWMutex unless reads dominate heavily and the critical section is long, because the RWMutex has bookkeeping of its own. sync.Map is for two narrow cases, keys written once and read many times, and goroutines working on disjoint keys; for everything else a map and a mutex is clearer and often faster, and the measurements are in the article. sync.Pool is an optimisation for reducing garbage, with no guarantee that anything stays in it, so never put anything in it that must not be lost. sync.Cond is rarely needed once you know channels and mutexes, and it is worth reading about mainly so you recognise it in other people's code. Atomics are for a single value, a flag, a counter or a pointer to an immutable snapshot. The moment you need to keep two values consistent with each other, you need a mutex.

Two cross-cutting rules for locks. Keep critical sections short and free of I/O, because everyone queued on the lock inherits the latency of whatever happens inside it. And never copy a struct that contains a mutex; go vet catches it, and you should let it.

The race detector is part of the toolchain

Whatever you choose, run the tests with go test -race. The race detector instruments memory accesses and reports two goroutines touching the same variable without synchronisation, with the stack of each. It finds real bugs, it has no false positives in the sense that matters, and the cost is a slower, more memory-hungry build that you run in CI. I treat a race report as a failing test, whatever the program appears to do, because a race that "works" on one machine is the kind that corrupts data on another. The articles linked in this hub use -race in their own test runs for that reason.

Context: cancellation and deadlines as part of the API

The context package is how a Go program says "stop" and "you have until then". Context explained is the guide. The convention is simple and strictly followed: the first parameter of any function that may block or take long is ctx context.Context, and the function returns promptly when ctx.Done() closes.

The pieces fit together like this. A request enters at the edge with a context that is cancelled when the client goes away. Cancelling work with context shows how to make that cancellation reach the database call and the downstream HTTP request. You derive a child with a deadline using WithTimeout and WithDeadline, and the child is cancelled when the deadline passes or when the parent is, whichever is first. Always call the cancel function, usually with defer, because otherwise the timer and the child stay alive until the parent ends.

Values on a context are the part of the package to be careful with. Context values explains the narrow set of things that belong there (request-scoped data such as a trace ID, which crosses API boundaries) and the larger set that does not (optional parameters, dependencies, anything the function needs to do its job). Newer additions fill gaps the original design left. Cancellation causes lets a cancel say why, so that a log can distinguish a shutdown from a timeout from a failed sibling, and AfterFunc and WithoutCancel cover cleanup that must run when a context ends and work that must outlive the request that started it, such as writing an audit record after the client has disconnected.

The error you will see most often is context deadline exceeded, and the one that confuses people is context canceled. The first says a time budget ran out, the second says someone cancelled it deliberately, and treating them the same hides the difference between a slow dependency and a client who closed the tab. Handling context canceled errors shows how to separate them in logs and in HTTP responses, and the guide to handling errors across the whole stack is Go error handling.

A rule I enforce in code review: a goroutine that is started with a context stops when that context is done, and a function that accepts a context but ignores it is a bug with a delayed fuse.

Leaks: the failure that does not crash

A goroutine leak is a goroutine that will never finish, usually blocked forever on a channel nobody will use or a lock nobody will release. Each one holds its stack and everything it references, and a service that leaks one per request runs out of memory at a speed proportional to its traffic, and does so quietly. Goroutine leaks is the guide to the common causes, which are a send with no receiver, a receive with no sender, a missing cancellation, a forgotten ticker and a worker with no exit path.

Detection comes in three layers. In production, watch the goroutine count: runtime.NumGoroutine shows how to export it and what a healthy graph looks like, which is flat or sawtooth, never a staircase. In tests, check that a test leaves nothing behind; goleak describes the library that does this, with the caveat that it reports leaks and does not stop them. In review, ask the question that prevents most leaks: how does this goroutine stop?

The answer should be one of a few shapes. Stopping a goroutine compares them. A context is the default. A done channel, closed by the owner, is the same idea without the package, and the done channel pattern shows it, including how to make a function that returns a channel safe to abandon. A closed input channel is how a pipeline stage learns that its upstream is finished. Whichever you pick, the stopping condition must be reachable from every blocking point inside the goroutine, which means each send and receive sits in a select with the cancellation case.

Worker pools and the pipelines that feed them

When there is more work than you want to run at once, you bound it. The worker pool is the oldest and most reliable way: N goroutines read jobs from a channel, do them, and write results to another channel. Building a worker pool is a full project with graceful shutdown, error collection and tests, and it is where I would send anyone who has only used go func() in a loop.

Choosing N is a measurement, not a guess. Sizing a worker pool benchmarks the same workload at different pool sizes for CPU-bound and I/O-bound jobs. The summary is that CPU-bound work stops improving near the number of cores, and I/O-bound work keeps improving until the downstream resource, a database pool or a rate limit, becomes the bottleneck. That second case is why the limit often belongs to the dependency, not to the code that calls it.

Pools compose with two patterns that describe how data flows. Fan-out and fan-in spreads work from one source across several workers and merges their results into one stream; fan-out, fan-in shows the merge with correct closing and cancellation. A pipeline is a chain of stages connected by channels, each stage doing one transformation, and the whole thing is cancelled by one context; the pipeline pattern builds one and, importantly, shows the stage that forgets to watch for cancellation and leaks when the consumer stops early. The pipeline article is one of the places where I rewrote my own first attempt after a test caught it, and it is the reason the section on leaks above is so insistent.

Errgroup and bounded concurrency

Most real fan-outs have an error story: if one of the ten calls fails, cancel the rest and return the error. That is what golang.org/x/sync/errgroup is for, and errgroup explains the semantics precisely, including the subtle point that only the first error is returned. Limiting errgroup concurrency adds SetLimit, which turns an errgroup into a bounded pool in four lines, and TryGo, which gives you a non-blocking submit that is useful for load shedding. When the work does not fit an errgroup, a semaphore built from a buffered channel or from x/sync/semaphore bounds concurrency in general, with weights if tasks differ in cost.

The error-handling side of goroutines has its own article, because a goroutine cannot return a value to its starter: returning errors from goroutines compares channels, errgroup and collecting into a slice, and joining several errors shows how to report all failures when the first is not enough.

Controlling the rate of work

A bound on concurrency limits how much runs at once. A rate limit controls how often it starts, which is a different requirement and often an externally imposed one: an API that allows 100 requests a second does not care how many are in flight. Rate limiting in Go compares the options, a ticker, x/time/rate and a hand-built limiter, and a token bucket limiter builds one from scratch and tests it against the library on 20,000 decisions, which is how I found a floating-point bug in my first version. Build your own to understand it, then use the library.

Two patterns reduce work rather than pace it. Singleflight collapses concurrent calls for the same key into one, which is the cheapest cache-stampede protection there is. Debounce and throttle turn a burst of events into one action, which matters for file watchers, search-as-you-type and anything driven by a noisy source.

Time: timers, tickers and schedules

Time-based concurrency has its own traps, and all of them have bitten me. A time.After in a loop used to allocate a timer per iteration that lived until it fired; the behaviour changed in Go 1.23, and the time.After leak explains what changed and what still needs care. A ticker must be stopped, and stopping tickers shows why a forgotten one is a leak you cannot see in the goroutine count. For a job that runs on a schedule, periodic tasks compares a sleep loop, a ticker and a re-armed timer, with measured drift, and shows how to avoid overlapping runs and add jitter.

Staying up when dependencies misbehave

The last group of patterns is about the world outside your process, which fails in ways your code cannot prevent. Each pattern answers a different failure and none replaces another.

• Retry with backoff and jitter handles transient failures without synchronising a fleet of clients into a thundering herd. Retrying with exponential backoff simulates the herd and the fix.
• A circuit breaker stops calling a dependency that is clearly down, so you fail fast and give it room to recover. The circuit breaker implements the three states and tests the single-probe half-open behaviour under concurrency.
• Bulkheads and load shedding keep one slow dependency from consuming all your capacity and keep the whole service healthy when traffic exceeds it. Load shedding and bulkheads measures both.

The ordering of these in a call matters. A typical outbound call is wrapped like this, from the outside in: a bulkhead to limit concurrent calls to this dependency, a circuit breaker to fail fast if it is down, a retry with backoff for transient errors, and a timeout from a context on each attempt. Reverse the retry and the breaker and every retry counts as a separate failure to the breaker, which opens too early. Put the retry outside the bulkhead and a retry storm can hold slots in the bulkhead while it waits.

Putting several patterns together

The patterns above are small on purpose. Here is how they combine into a function I use constantly: apply a function to every element of a slice, with a bound on concurrency, results in input order, and cancellation of the remaining work as soon as one call fails. It uses a semaphore for the bound, a WaitGroup for the wait, a context with a cause for the cancellation, and a sync.Once so only the first error is recorded.

package main

import (
	"context"
	"sync"
)

// Map applies fn to every element of in with at most limit calls running at
// once. Results keep the input order. After the first error the context given
// to the remaining calls is cancelled and no new call is started.
func Map[In, Out any](ctx context.Context, limit int, in []In, fn func(context.Context, In) (Out, error)) ([]Out, error) {
	ctx, cancel := context.WithCancelCause(ctx)
	defer cancel(nil)

	out := make([]Out, len(in))
	sem := make(chan struct{}, limit)

	var (
		wg    sync.WaitGroup
		once  sync.Once
		first error
	)

loop:
	for i, v := range in {
		if ctx.Err() != nil {
			break
		}
		select {
		case sem <- struct{}{}:
		case <-ctx.Done():
			break loop
		}
		wg.Go(func() {
			defer func() { <-sem }()
			res, err := fn(ctx, v)
			if err != nil {
				once.Do(func() {
					first = err
					cancel(err)
				})
				return
			}
			out[i] = res
		})
	}
	wg.Wait()

	if first != nil {
		return nil, first
	}
	return out, ctx.Err()
}

The loop acquires a slot before it starts each goroutine, so the number of goroutines alive never exceeds limit and the producer is the one that waits, which is the backpressure you want. Each goroutine writes only to its own index of out, so there is no race on the slice and no lock is needed. The check of ctx.Err() at the top of each iteration makes the stop deterministic, because when both the semaphore and the done channel are ready, select chooses at random. The loop variables are per-iteration since Go 1.22, so capturing i and v in the closure is safe. A parent context cancelled from outside surfaces as the returned error through ctx.Err().

A run shows the three behaviours:

package main

import (
	"context"
	"errors"
	"fmt"
	"sync/atomic"
	"time"
)

func main() {
	ctx := context.Background()

	// 1. Ordered results with a bound of 3.
	var running, peak atomic.Int32
	nums := []int{1, 2, 3, 4, 5, 6, 7, 8}
	squares, err := Map(ctx, 3, nums, func(_ context.Context, n int) (int, error) {
		cur := running.Add(1)
		for {
			p := peak.Load()
			if cur <= p || peak.CompareAndSwap(p, cur) {
				break
			}
		}
		time.Sleep(10 * time.Millisecond)
		running.Add(-1)
		return n * n, nil
	})
	fmt.Println("results:", squares, "err:", err)
	fmt.Println("peak concurrency is at most 3:", peak.Load() <= 3)

	// 2. The first error stops the rest.
	var started atomic.Int32
	boom := errors.New("boom")
	_, err = Map(ctx, 2, make([]int, 20), func(ctx context.Context, _ int) (int, error) {
		n := started.Add(1)
		if n == 3 {
			return 0, boom
		}
		select {
		case <-time.After(20 * time.Millisecond):
			return 0, nil
		case <-ctx.Done():
			return 0, ctx.Err()
		}
	})
	fmt.Println("error is boom:", errors.Is(err, boom))
	fmt.Println("stopped early, started fewer than 20:", started.Load() < 20)

	// 3. A cancelled parent surfaces as the error.
	cctx, cancel := context.WithCancel(ctx)
	cancel()
	_, err = Map(cctx, 2, nums, func(context.Context, int) (int, error) { return 0, nil })
	fmt.Println("cancelled parent:", err)
}

results: [1 4 9 16 25 36 49 64] err: <nil>
peak concurrency is at most 3: true
error is boom: true
stopped early, started fewer than 20: true
cancelled parent: context canceled

The tests pin the same properties so a future edit cannot break them quietly:

package main

import (
	"context"
	"errors"
	"sync/atomic"
	"testing"
	"time"
)

func TestMapKeepsInputOrder(t *testing.T) {
	in := []int{5, 4, 3, 2, 1}
	got, err := Map(context.Background(), 4, in, func(_ context.Context, n int) (int, error) {
		time.Sleep(time.Duration(n) * time.Millisecond) // later items finish first
		return n * 10, nil
	})
	if err != nil {
		t.Fatal(err)
	}
	for i, n := range in {
		if got[i] != n*10 {
			t.Fatalf("got[%d] = %d, want %d", i, got[i], n*10)
		}
	}
}

func TestMapNeverExceedsLimit(t *testing.T) {
	var running, peak atomic.Int32
	_, err := Map(context.Background(), 3, make([]int, 30), func(context.Context, int) (int, error) {
		cur := running.Add(1)
		for {
			p := peak.Load()
			if cur <= p || peak.CompareAndSwap(p, cur) {
				break
			}
		}
		time.Sleep(2 * time.Millisecond)
		running.Add(-1)
		return 0, nil
	})
	if err != nil {
		t.Fatal(err)
	}
	if peak.Load() > 3 {
		t.Fatalf("peak concurrency %d exceeds the limit of 3", peak.Load())
	}
}

func TestMapReturnsFirstErrorAndStops(t *testing.T) {
	sentinel := errors.New("first")
	var started atomic.Int32
	_, err := Map(context.Background(), 2, make([]int, 50), func(ctx context.Context, _ int) (int, error) {
		if started.Add(1) == 2 {
			return 0, sentinel
		}
		<-ctx.Done()
		return 0, ctx.Err()
	})
	if !errors.Is(err, sentinel) {
		t.Fatalf("err = %v, want the first error", err)
	}
	if started.Load() > 4 {
		t.Fatalf("%d calls started after the failure; the loop should have stopped", started.Load())
	}
}

func TestMapHonoursCancelledParent(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	cancel()
	var calls atomic.Int32
	_, err := Map(ctx, 2, make([]int, 10), func(context.Context, int) (int, error) {
		calls.Add(1)
		return 0, nil
	})
	if !errors.Is(err, context.Canceled) {
		t.Fatalf("err = %v, want context.Canceled", err)
	}
	if calls.Load() != 0 {
		t.Fatalf("%d calls ran with an already cancelled context", calls.Load())
	}
}

=== RUN   TestMapKeepsInputOrder
--- PASS: TestMapKeepsInputOrder (0.01s)
=== RUN   TestMapNeverExceedsLimit
--- PASS: TestMapNeverExceedsLimit (0.03s)
=== RUN   TestMapReturnsFirstErrorAndStops
--- PASS: TestMapReturnsFirstErrorAndStops (0.00s)
=== RUN   TestMapHonoursCancelledParent
--- PASS: TestMapHonoursCancelledParent (0.00s)
PASS
ok  	example.test/lab	1.284s

A function like this is worth writing once and testing hard. When you reach for go func() in a loop, the question to ask is whether this helper already does what you need.

A word on the version this was tested with. Several of the simplifications above depend on recent releases: per-iteration loop variables (Go 1.22), the reworked timers (Go 1.23) and WaitGroup.Go (Go 1.25). If your module targets an older Go, the go line in go.mod decides which semantics you get, so check it before assuming a snippet from this site behaves the way it did for me.

Choosing a pattern: a decision table

When I am staring at a concurrency problem, I run through this list in order and stop at the first row that fits.

If none of the rows fit, I try to reduce the problem to one that does before I invent a new coordination scheme. A custom synchronisation protocol built on three channels and a mutex works on the day you write it, and it is nearly impossible for the next reader to verify.

Testing and debugging concurrent code

Concurrent code needs the tests that sequential code can skip. The checklist I apply to a new component:

1. Run with -race and run many times. go test -race -count=100 finds flaky races that a single pass hides.
2. Test cancellation explicitly. Cancel mid-flight and assert that the function returns promptly and leaves nothing running. The pipeline and worker pool articles show tests that do this with a stalled source.
3. Avoid sleeps as synchronisation. A test that sleeps to wait for a goroutine is slow when it works and flaky when it does not. Use channels to know when something has happened, a fake clock for time-based logic (the circuit breaker and rate limiter articles inject one), and a generous timeout only as a safety net to fail the test instead of hanging.
4. Check for leaks. Compare the goroutine count before and after, or use goleak, in the tests of anything that starts a goroutine.
5. Test the error paths. The goroutine that fails halfway is the one that exposes a missing Done.

When a running program misbehaves, the tools are built in. A SIGQUIT dumps every goroutine's stack, and the goroutine profile at /debug/pprof/goroutine?debug=2 gives the same view from a live service, with the blocked state and the wait time of each. The execution tracer shows scheduling, blocking and GC on a timeline, and is the right tool when the question is why a pool of eight workers uses two cores. Reading those outputs is a skill that pays back quickly: "a thousand goroutines blocked on the same line" is almost always the whole diagnosis.

Production checklist

Before a concurrent component goes to production, I want a yes to each of these:

• Every goroutine has an owner, and I can say where it is waited for.
• Every blocking call is cancellable, through a context or a done channel.
• Every pool, queue, buffer and retry loop has a bound, and the bound is a number somebody chose.
• A panic in a goroutine cannot take the process down unobserved: the entry point recovers, logs with the stack and either restarts the work or fails the request.
• Shutdown is graceful: stop accepting work, let in-flight work finish within a deadline, then force.
• The component exports its queue depth, in-flight count, rejections and the age of the oldest item.
• A test covers cancellation and a test covers the error path.
• Under twice the expected load it degrades by rejecting work, not by slowing everything.

The last item is the one that matters most and gets checked least. A load test at 150 percent of capacity that asserts the latency of the accepted requests stays within budget is the single best piece of evidence that the system is built correctly. The articles on load shedding, circuit breakers and rate limiting are about making that test pass.

Where to go next

If you are new to the cluster, read in this order: goroutines, channels, select, WaitGroup and Mutex, then context, then goroutine leaks. That is the foundation, and everything else builds on it. Then build the worker pool, which exercises all of it at once, and the pipeline, which is the best teacher of cancellation. The pattern articles after that are reference material, read when the problem arrives.

For the primary sources, the Go team's own writing is the best and most patient: the Go Concurrency Patterns: Pipelines and cancellation post, the Go Concurrency Patterns: Context post, the Go memory model, the Effective Go section on concurrency, and the package documentation for sync, sync/atomic and context. For the system-level view of why bounds, shedding and backoff matter, the Google SRE book's chapter on handling overload is the one I reread most.

If a part of this hub is thin for what you are building, the related guides on Go error handling and the performance articles pick up where concurrency leaves off, since a program that handles failure well and uses its goroutines sparingly is usually a fast one as well.