Bulkheads and Load Shedding in Go Services Under Pressure

Every service has a capacity, and every service eventually gets more traffic than that. What happens next separates systems that stay up from systems that fall over. The usual outcome without preparation is not a graceful slowdown. It is a collapse: requests queue, the queue adds latency, clients time out and retry (adding load), and the service spends its capacity on responses that nobody is waiting for any more. The throughput of useful work falls to zero while the CPUs are busy.

This guide explains why that happens, then shows the two defences, load shedding and bulkheads, with code and measurements. It builds on semaphores, rate limiting, circuit breakers and context deadlines. Everything ran on Go 1.25.6.

Why overload collapses

A server that can handle 100 requests a second and receives 150 does not serve 100 and drop 50. Without any limit, it accepts all 150, and starts all of them. Each is now slower, because they share the CPU and the connections. Latency rises, and when it exceeds the clients' timeout, the clients give up. But the server is still working on those requests, so its capacity is spent producing answers into closed connections, and the clients retry, adding load. The mechanism is queueing: a system running above its capacity builds an unbounded queue, and the wait in the queue is the latency.

Queueing theory has a formula for the simplest case: as utilisation approaches 100%, the average wait grows without limit, and above 100% it grows forever. The practical lesson is that the response to overload must be to refuse work, because the alternative is to do all of it badly. A request refused in a millisecond costs the client a retry or an error message. A request accepted and abandoned after thirty seconds costs the client the wait and the server the work.

Shedding: the simplest version

A load shedder admits a bounded number of requests at once and rejects the rest immediately. The bound is a semaphore with a non-blocking acquire (semaphores), and the rejection is a 503 with a Retry-After header. Here is the effect, measured. A server with room for 4 concurrent requests, each taking 50 ms, is hit by 40 clients at once. First without a limit, then with one:

package main

import (
	"net/http"
	"time"
)

// Shed admits at most limit requests at once. A request that cannot get a slot
// within wait is rejected with 503 and a Retry-After header.
func Shed(limit int, wait time.Duration, next http.Handler) http.Handler {
	slots := make(chan struct{}, limit)
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		select {
		case slots <- struct{}{}:
		default:
			// no free slot: wait briefly, then give up
			t := time.NewTimer(wait)
			defer t.Stop()
			select {
			case slots <- struct{}{}:
			case <-t.C:
				w.Header().Set("Retry-After", "1")
				http.Error(w, "overloaded", http.StatusServiceUnavailable)
				return
			case <-r.Context().Done():
				return
			}
		}
		defer func() { <-slots }()
		next.ServeHTTP(w, r)
	})
}

Shed wraps any handler. A free slot is taken at once. With none free, the request waits up to wait for one, and if the wait expires it is rejected, so the queue has a bounded length and a bounded time: at most limit requests run, and a request never waits longer than wait before knowing its fate. A client that disconnects while waiting is dropped without being counted. The deferred receive frees the slot on every exit path.

Now compare the two servers under the same load:

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"sort"
	"sync"
	"time"
)

func work(w http.ResponseWriter, r *http.Request) {
	time.Sleep(50 * time.Millisecond) // the service's real work
	fmt.Fprint(w, "ok")
}

// slots simulates limited capacity: only 4 requests make progress at once.
func capacityLimited(next http.HandlerFunc) http.Handler {
	cpu := make(chan struct{}, 4)
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		cpu <- struct{}{} // queues inside the server when capacity is exceeded
		defer func() { <-cpu }()
		next(w, r)
	})
}

type result struct {
	status  int
	latency time.Duration
}

func hammer(url string, clients int) []result {
	var (
		wg  sync.WaitGroup
		mu  sync.Mutex
		out []result
	)
	for range clients {
		wg.Go(func() {
			start := time.Now()
			resp, err := http.Get(url)
			if err != nil {
				return
			}
			resp.Body.Close()
			mu.Lock()
			out = append(out, result{resp.StatusCode, time.Since(start)})
			mu.Unlock()
		})
	}
	wg.Wait()
	return out
}

func summarise(name string, rs []result) {
	var ok []time.Duration
	rejected := 0
	for _, r := range rs {
		if r.status == 200 {
			ok = append(ok, r.latency)
		} else {
			rejected++
		}
	}
	sort.Slice(ok, func(i, j int) bool { return ok[i] < ok[j] })
	slowest := time.Duration(0)
	if len(ok) > 0 {
		slowest = ok[len(ok)-1]
	}
	fmt.Printf("%-12s served=%2d rejected=%2d slowest served request about %v\n",
		name, len(ok), rejected, slowest.Round(50*time.Millisecond))
}

func main() {
	unlimited := httptest.NewServer(capacityLimited(work))
	defer unlimited.Close()
	shedding := httptest.NewServer(Shed(4, 10*time.Millisecond, capacityLimited(work)))
	defer shedding.Close()

	summarise("no shedding", hammer(unlimited.URL, 40))
	summarise("with shedding", hammer(shedding.URL, 40))
}

no shedding  served=40 rejected= 0 slowest served request about 500ms
with shedding served= 4 rejected=36 slowest served request about 50ms

Forty clients arrive together at a server that does four requests at a time, each taking 50 ms. Without shedding, all forty are accepted and queue inside the server: ten rounds of four, so the last client waits about 500 ms. With the shedder, the four admitted requests finish in about 50 ms, a few more get in as slots free during the 10 ms bounded wait, and the rest are turned away immediately with a 503. The slowest served request stays close to the 50 ms the work takes, instead of ten times that. The rejected requests got an answer at once, which a well-behaved client treats as a signal to back off (retry with backoff).

In production the comparison is starker, because clients have timeouts. A client that gives up at 300 ms gets nothing from the unlimited server for most of the forty requests, while the server wastes its capacity finishing them, and gets fast, correct failures from the shedding one.

What to shed, and when

A fixed concurrency limit is the simplest policy and a good start. A few refinements are worth knowing.

Choose the limit from capacity. The right number is what the service can run concurrently without latency degrading, usually found by load testing: raise concurrency until throughput stops improving or latency starts to climb, and set the limit near the knee (pool sizing describes the measurements). Too low wastes capacity, too high stops protecting.

Bound the wait. A queue in front of the limit is useful for absorbing short bursts, and harmful if it is deep. A wait of a few milliseconds up to the typical service time is enough to smooth jitter; beyond that you are adding latency the client will not tolerate. Make the wait shorter than the client's timeout by a wide margin.

Shed early. Reject as close to the front as possible, before parsing, authentication and database calls. A rejected request should cost almost nothing. A middleware at the top of the stack, or the load balancer, is the place.

Use the deadline. If a request arrives with a deadline that has nearly passed, or has been waiting so long that its deadline cannot be met, drop it: nobody is waiting for the answer. Check ctx.Err() before starting expensive work (cancellation in web services).

Signal clearly. 503 Service Unavailable with Retry-After is the standard, 429 Too Many Requests when it is a per-client limit (rate limiting). Make the body short and cheap to produce.

Bulkheads: isolate the dependencies

A ship's bulkheads divide the hull into watertight compartments so that a breach floods one and not all. In a service, the equivalent is giving each dependency (or each class of work) its own limit, so a problem with one cannot consume the shared resource, the goroutines, connections or threads, that everything else needs.

The failure it prevents is easy to produce. A service calls two dependencies from one shared pool of workers: a fast one that answers in 5 ms and a slow one that starts hanging. Requests to the slow one pile up and occupy every worker, and the fast one, which is perfectly healthy, can no longer get a worker. One sick dependency has taken down the whole service.

package main

import (
	"fmt"
	"sync"
	"time"
)

// pool runs jobs on a fixed set of workers, standing in for any bounded resource.
type pool struct{ slots chan struct{} }

func newPool(n int) *pool { return &pool{slots: make(chan struct{}, n)} }

func (p *pool) run(job func()) {
	p.slots <- struct{}{}
	defer func() { <-p.slots }()
	job()
}

func fastCall() { time.Sleep(5 * time.Millisecond) }
func slowCall() { time.Sleep(300 * time.Millisecond) } // a dependency that has started hanging

// measure sends slowN slow calls and then fastN fast calls through the pools
// and reports how long the fast calls took in total.
func measure(slowPool, fastPool *pool, slowN, fastN int) time.Duration {
	var wg sync.WaitGroup
	for range slowN {
		wg.Go(func() { slowPool.run(slowCall) })
	}
	time.Sleep(20 * time.Millisecond) // let the slow calls take their slots

	start := time.Now()
	var fast sync.WaitGroup
	for range fastN {
		fast.Go(func() { fastPool.run(fastCall) })
	}
	fast.Wait()
	elapsed := time.Since(start)
	wg.Wait()
	return elapsed
}

func main() {
	shared := newPool(4)
	fmt.Println("shared pool:    healthy calls took about",
		measure(shared, shared, 4, 8).Round(50*time.Millisecond), "behind four hung calls")

	slowPool, fastPool := newPool(2), newPool(2) // separate compartments
	fmt.Println("with bulkheads: healthy calls took about",
		measure(slowPool, fastPool, 4, 8).Round(10*time.Millisecond), "while the slow dependency hangs")
}

shared pool:    healthy calls took about 300ms behind four hung calls
with bulkheads: healthy calls took about 20ms while the slow dependency hangs

With one shared pool of four, four hung calls take every slot for 300 ms, and eight healthy calls that should take 20 ms wait behind them: they finish in about 300 ms plus. With separate pools, the slow dependency can use only its two slots (its excess queues in its own compartment), and the healthy dependency's calls complete in about 20 ms, untouched. The cost is that the total capacity is divided, so the slow dependency is limited even when the fast one is idle. That is the trade: isolation in exchange for flexibility. In practice, size each bulkhead from that dependency's own capacity and importance, and let the smaller ones be small.

Bulkheads can be pools of goroutines, semaphores, separate HTTP clients with their own connection limits (Transport.MaxConnsPerHost), separate database pools, or separate processes. The principle is the same at each level: do not let the failure domain of one thing be the capacity of everything.

Where the limit lives: process, proxy or both

A shedder in the Go process is the last line of defence, and it has one weakness: by the time a request reaches your handler, the server has already accepted the connection, read the headers and spent a goroutine on it. Under a flood, that work is not free. A shedder at the load balancer or reverse proxy turns the request away before it costs the application anything, and most proxies (nginx, Envoy, HAProxy) can cap concurrent connections or requests per upstream. The two layers complement each other. The proxy protects the fleet from a surge, using a coarse limit. The in-process limit protects each instance from its own particular bottleneck, such as a database pool of 20 connections that no proxy knows about.

Keep the in-process limit consistent with the resource behind it. If a handler needs one database connection and the pool has 20, a concurrency limit of 200 does nothing useful: 180 requests will be waiting for a connection inside the handler, holding memory and a goroutine, with no bounded wait and no 503. Set the limit at or slightly above the pool size, so the queue lives in front of the handler where you control it, not inside it where you do not.

A fixed limit versus an adaptive one

A hand-tuned limit goes stale. The number that was right last quarter is wrong after a new feature makes each request heavier, or after a dependency gets faster. Adaptive limiters adjust the limit from observed latency, borrowing the idea from TCP congestion control: grow the limit slowly while latency stays near its floor, and cut it sharply when latency rises. Netflix's concurrency-limits library popularised the approach with AIMD and gradient algorithms, and the same loop is a few dozen lines in Go. The cost is that it is harder to reason about, and a noisy latency signal can make the limit oscillate. A sensible path is to start with a fixed limit from a load test, export the in-flight and rejection metrics, and move to an adaptive one only when you see the fixed number drifting out of date. If you do adapt, bound it with a floor and a ceiling so a bad measurement cannot set the limit to zero or to infinity.

What the client should do with a 503

Shedding only works if clients cooperate. A client that retries a 503 immediately, in a tight loop, turns your protection into an attack on yourself. Document the contract: honour Retry-After, back off exponentially with jitter (retry with backoff), cap the number of attempts, and stop retrying altogether when a circuit breaker is open. For internal clients you control, enforce this in the shared HTTP client so nobody writes a naive loop. For browsers and third parties you cannot, the rate limiter at the edge (rate limiting) is what keeps one badly behaved caller from using the entire admitted capacity.

Protecting critical traffic

Not all requests are equally valuable. Health checks, payments and logins matter more than a recommendations widget or a background export. Two techniques keep the important work flowing under load. Reserve capacity: give critical traffic its own bulkhead, or let it use the whole limit while best-effort traffic can use only a fraction (for example, best-effort is shed when more than 70% of slots are taken). Shed by priority: when the limiter is full, reject low-priority requests first. A small extension of the shedder does this by checking occupancy against a per-class threshold:

package main

import (
	"net/http"
	"net/http/httptest"
	"sync"
	"testing"
	"time"
)

// priorityShed lets critical requests use every slot but best-effort ones only
// a fraction of them.
func priorityShed(limit, bestEffortLimit int, next http.Handler) http.Handler {
	var mu sync.Mutex
	inFlight := 0
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		critical := r.Header.Get("X-Priority") == "critical"
		mu.Lock()
		cap := bestEffortLimit
		if critical {
			cap = limit
		}
		if inFlight >= cap {
			mu.Unlock()
			w.Header().Set("Retry-After", "1")
			http.Error(w, "overloaded", http.StatusServiceUnavailable)
			return
		}
		inFlight++
		mu.Unlock()
		defer func() { mu.Lock(); inFlight--; mu.Unlock() }()
		next.ServeHTTP(w, r)
	})
}

func TestCriticalTrafficKeepsCapacityUnderLoad(t *testing.T) {
	release := make(chan struct{})
	entered := make(chan struct{}, 10)
	slow := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		entered <- struct{}{}
		<-release
	})
	h := priorityShed(5, 3, slow)

	// Three best-effort requests fill the best-effort share.
	var wg sync.WaitGroup
	for range 3 {
		wg.Go(func() {
			h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
		})
	}
	for range 3 {
		<-entered
	}

	best := httptest.NewRecorder()
	h.ServeHTTP(best, httptest.NewRequest("GET", "/", nil))
	if best.Code != http.StatusServiceUnavailable {
		t.Errorf("a fourth best-effort request got %d, want 503", best.Code)
	}

	crit := httptest.NewRequest("GET", "/", nil)
	crit.Header.Set("X-Priority", "critical")
	done := make(chan int, 1)
	go func() {
		rec := httptest.NewRecorder()
		h.ServeHTTP(rec, crit)
		done <- rec.Code
	}()
	select {
	case <-entered: // the critical request was admitted into the reserved capacity
	case code := <-done:
		t.Fatalf("the critical request was rejected with %d; it should use the reserved slots", code)
	case <-time.After(time.Second):
		t.Fatal("the critical request never started")
	}

	close(release)
	wg.Wait()
}

=== RUN   TestCriticalTrafficKeepsCapacityUnderLoad
--- PASS: TestCriticalTrafficKeepsCapacityUnderLoad (0.00s)
PASS
ok  	example.test/lab	1.277s

Best-effort traffic fills its three slots, a fourth best-effort request is turned away, and a critical request still gets in, because two of the five slots are reserved for it. The test uses channels to hold the requests in flight deterministically, so it needs no sleeps. The reservation is the whole mechanism: priority is a threshold on shared occupancy, not a separate queue.

Measuring it

Shedding that you cannot see is shedding you cannot tune. Export four numbers: requests rejected (by reason and class), in-flight requests, queue wait time for those that waited, and the latency of accepted requests. A healthy shedder shows rejections that rise during load while the latency of accepted requests stays flat; a limit that is too low shows rejections at normal load, and one that is too high shows accepted-request latency rising with load. Alert on a sustained rejection rate, not on the first rejection: shedding is the system working. Test it under load before you need it: a load test that runs at 150% of capacity and checks that the 99th percentile of accepted requests stays within its target is the proof that the mechanism works.

Putting the pieces together

The defences stack, and each answers a different failure. A timeout on each call bounds the cost of one slow request. A retry with backoff and jitter handles brief failures without synchronising clients. A circuit breaker stops calling a dependency that is clearly down (circuit breakers). A bulkhead isolates the failure domain. A load shedder at the front protects the service itself from more work than it can do. None replaces another. A service that has only retries makes an overload worse; one that has only shedding still fails when a dependency hangs. In a design review, ask for each dependency: what happens when it is slow, when it is down, and when traffic doubles, and make sure the answer is "this limit trips and this alert fires".

For further reading, the Google SRE book's chapters on handling overload and addressing cascading failures describe these ideas at scale, and the building blocks used here are in sync, net/http and context.