Build a Load Balancer in Go With Active Health Checks

A load balancer is a reverse proxy that can choose. Everything that makes it hard is in the choosing: which backend gets this request, how you know a backend is dead, how you know it has come back, and what you do in the gap between the backend dying and your noticing. The proxy half is already solved by httputil.ReverseProxy, so this project spends its time on the policy.

It builds on the reverse proxy article, which covers Rewrite, error handling and the transport settings a balancer inherits, and it belongs to the web cluster that ends in a production HTTP server. The health probing relies on the timeout discipline from the HTTP client article. Everything ran on Go 1.25.6 with the standard library only.

The design

Three types and one goroutine:

• A Backend has a URL, its own ReverseProxy, an alive flag and an inflight counter. Both are atomics, because the request path reads them on every request and the checker writes them from another goroutine.
• A Pool holds the backends and a counter for round robin. pick returns the next live backend or nil.
• A health checker probes each backend on an interval and flips alive based on thresholds.
• The pool is an http.Handler, so it plugs in anywhere a handler does, including behind the middleware from the middleware article.

I deliberately avoid a mutex around the backend list. The list never changes after construction here; if you need to add and remove backends at runtime, swap the whole slice atomically with atomic.Pointer[[]*Backend] instead of locking the hot path.

The pool

package main

import (
	"context"
	"net/http"
	"net/http/httputil"
	"net/url"
	"strconv"
	"sync"
	"sync/atomic"
	"time"
)

type Backend struct {
	URL      *url.URL
	proxy    *httputil.ReverseProxy
	alive    atomic.Bool
	inflight atomic.Int64
	fails    int // touched only by the checker goroutine
	passes   int
}

type Pool struct {
	backends   []*Backend
	next       atomic.Uint64
	leastConn  bool
	rise, fall int
}

func NewPool(urls []string, leastConn bool) *Pool {
	p := &Pool{leastConn: leastConn, rise: 2, fall: 2}
	for _, raw := range urls {
		u, err := url.Parse(raw)
		if err != nil {
			panic(err)
		}
		b := &Backend{URL: u}
		b.alive.Store(true)
		b.proxy = &httputil.ReverseProxy{
			Rewrite: func(pr *httputil.ProxyRequest) {
				pr.SetURL(u)
				pr.SetXForwarded()
			},
			Transport: &http.Transport{MaxIdleConnsPerHost: 64, ResponseHeaderTimeout: 2 * time.Second},
			ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
				b.alive.Store(false) // passive check: the next request will not pick this backend
				http.Error(w, "bad gateway", http.StatusBadGateway)
			},
		}
		p.backends = append(p.backends, b)
	}
	return p
}

// pick returns a live backend, or nil when none is left.
func (p *Pool) pick() *Backend {
	n := len(p.backends)
	if p.leastConn {
		var best *Backend
		for _, b := range p.backends {
			if b.alive.Load() && (best == nil || b.inflight.Load() < best.inflight.Load()) {
				best = b
			}
		}
		return best
	}
	for i := 0; i < n; i++ {
		b := p.backends[(p.next.Add(1)-1)%uint64(n)]
		if b.alive.Load() {
			return b
		}
	}
	return nil
}

func (p *Pool) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	b := p.pick()
	if b == nil {
		w.Header().Set("Retry-After", strconv.Itoa(5))
		http.Error(w, "no healthy backend", http.StatusServiceUnavailable)
		return
	}
	b.inflight.Add(1)
	defer b.inflight.Add(-1)
	b.proxy.ServeHTTP(w, r)
}

// Check probes every backend once. Call it from a ticker; thresholds stop a single bad probe from flapping state.
func (p *Pool) Check(ctx context.Context, client *http.Client) {
	var wg sync.WaitGroup
	for _, b := range p.backends {
		wg.Add(1)
		go func() {
			defer wg.Done()
			ctx, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
			defer cancel()
			req, _ := http.NewRequestWithContext(ctx, http.MethodGet, b.URL.String()+"/healthz", nil)
			res, err := client.Do(req)
			ok := err == nil && res.StatusCode == http.StatusOK
			if res != nil {
				res.Body.Close()
			}
			if ok {
				b.fails, b.passes = 0, b.passes+1
				if b.passes >= p.rise {
					b.alive.Store(true)
				}
			} else {
				b.passes, b.fails = 0, b.fails+1
				if b.fails >= p.fall {
					b.alive.Store(false)
				}
			}
		}()
	}
	wg.Wait()
}

// Run probes on an interval until ctx ends.
func (p *Pool) Run(ctx context.Context, every time.Duration) {
	t := time.NewTicker(every)
	defer t.Stop()
	client := &http.Client{}
	for {
		select {
		case <-ctx.Done():
			return
		case <-t.C:
			p.Check(ctx, client)
		}
	}
}

Some choices in there deserve a line each. pick for round robin increments a shared atomic and skips dead entries, so a dead backend does not stall the rotation; the cost is that a dead one makes its neighbour take double its share, which is fine for a rotation of equal machines. leastConn scans for the lowest inflight, reading atomics without a lock: the answer can be a microsecond stale, and nobody cares. The Check function is exposed on its own so tests (and the demo below) can drive it step by step, without waiting for a ticker. And the error handler flips alive before writing the 502, so the very next request already avoids the dead backend instead of hitting it again while waiting for a probe.

The thresholds, rise and fall, are the part people skip. A backend that misses one probe because of a garbage collection pause or a network blip should stay in rotation; a backend that fails twice in a row should go. Likewise it should earn its way back with two passes before it takes traffic again, because a service that is restarting often answers its health endpoint for a moment before it can serve real requests.

Watching it work

The demo starts three backends that answer with their name, plus a health endpoint each can be switched off. It sends thirty requests at a time and counts where they land.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"sort"
	"sync/atomic"
)

type fake struct {
	name    string
	healthy atomic.Bool
	srv     *httptest.Server
}

func newFake(name string) *fake {
	f := &fake{name: name}
	f.healthy.Store(true)
	f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.URL.Path == "/healthz" {
			if !f.healthy.Load() {
				http.Error(w, "sick", http.StatusServiceUnavailable)
			}
			return
		}
		fmt.Fprint(w, f.name)
	}))
	return f
}

func spread(front *httptest.Server, n int) string {
	counts := map[string]int{}
	for i := 0; i < n; i++ {
		res, err := http.Get(front.URL)
		if err != nil {
			counts["error"]++
			continue
		}
		body, _ := io.ReadAll(res.Body)
		res.Body.Close()
		if res.StatusCode != 200 {
			counts[fmt.Sprintf("HTTP %d", res.StatusCode)]++
		} else {
			counts[string(body)]++
		}
	}
	keys := make([]string, 0, len(counts))
	for k := range counts {
		keys = append(keys, k)
	}
	sort.Strings(keys)
	out := ""
	for _, k := range keys {
		out += fmt.Sprintf(" %s=%d", k, counts[k])
	}
	return out
}

func main() {
	a, b, c := newFake("A"), newFake("B"), newFake("C")
	defer a.srv.Close()
	defer b.srv.Close()

	pool := NewPool([]string{a.srv.URL, b.srv.URL, c.srv.URL}, false)
	front := httptest.NewServer(pool)
	defer front.Close()
	ctx := context.Background()
	hc := &http.Client{}

	fmt.Println("all healthy:        ", spread(front, 30))

	b.healthy.Store(false)
	pool.Check(ctx, hc)
	fmt.Println("B fails one probe:  ", spread(front, 30))
	pool.Check(ctx, hc)
	fmt.Println("B fails two probes: ", spread(front, 30))

	b.healthy.Store(true)
	pool.Check(ctx, hc)
	fmt.Println("B passes one probe: ", spread(front, 30))
	pool.Check(ctx, hc)
	fmt.Println("B passes two probes:", spread(front, 30))

	c.srv.Close() // C dies without warning; no probe has run yet
	fmt.Println("C crashes:          ", spread(front, 30))

	a.srv.Close()
	b.srv.Close()
	fmt.Println("A and B crash too:  ", spread(front, 6))

	// The least-connections rule, with in-flight counts set by hand.
	lc := NewPool([]string{"http://a", "http://b", "http://c"}, true)
	lc.backends[0].inflight.Store(7)
	lc.backends[1].inflight.Store(2)
	lc.backends[2].inflight.Store(5)
	fmt.Println("least conn picks:   ", lc.pick().URL.Host)
	lc.backends[1].alive.Store(false)
	fmt.Println("...with b down:     ", lc.pick().URL.Host)
}

all healthy:          A=10 B=10 C=10
B fails one probe:    A=10 B=10 C=10
B fails two probes:   A=15 C=15
B passes one probe:   A=15 C=15
B passes two probes:  A=10 B=10 C=10
C crashes:            A=15 B=14 HTTP 502=1
A and B crash too:    HTTP 502=2 HTTP 503=4
least conn picks:    b
...with b down:      c

Walk through it from the top. With all three healthy, round robin splits thirty requests ten each. When B fails a single probe, nothing changes, which is the fall threshold doing its job; after the second failure B leaves the rotation and A and C take fifteen each. A single passing probe does not bring B back either, and the second does. That symmetry is what keeps a half-broken backend from bouncing in and out of service.

Then the interesting case: C crashes with no warning. No health probe has noticed, so the first request that round robin routes to C fails with a 502, the ErrorHandler marks C dead, and from then on the rotation skips it. One request lost, not a third of all traffic until the next probe, and the interval of your active checks no longer sets how long failures last. When A and B go as well, the first request to each fails with a 502 (the two in the output) and marks it down, after which pick returns nil and the pool answers the remaining four with 503 Service Unavailable and Retry-After, which is the honest response and the one clients know how to back off from.

The last two lines exercise pick without any network. Least connections chooses b (2 in flight), and when that one is marked down it falls to c (5). Unit tests for a picker should look like this: construct the state, call the function, assert.

Retrying, and why this balancer does not

The lost request in the crash case is the weak spot. A real balancer can retry it on another backend, and that is where the correctness traps are. Retry only when the failure happened before the backend could have processed the request (a refused or timed-out connection), only for methods that are safe to repeat (GET, HEAD, and PUT or DELETE when your handlers are idempotent), and only if the request body can be replayed, which for a streamed body it cannot. Retrying a POST that reached a backend which then crashed can create a duplicate order. The tools for doing it safely are idempotency keys, and the discipline from retries with backoff in the HTTP client.

If you want retries here, wrap the pool's Transport in a RoundTripper that picks a backend per attempt, rather than choosing before the proxy runs. That moves the decision into the one place that sees each attempt's outcome.

Sticky routing with a hash ring

Some traffic should not be spread evenly. A cache that holds per-user data in memory works best if a user's requests keep landing on the same backend, and a WebSocket hub per room wants every member of the room on one process. The obvious way is hash(key) % number_of_backends, and its flaw shows up the day a backend leaves: the divisor changes, and most keys map somewhere new at once, so every cache goes cold together. A hash ring fixes it. Each backend is placed at many points (virtual nodes) on a circle of hash values, a key is hashed onto the same circle, and it belongs to the first node clockwise from it. Remove a backend and only the keys that were on it move.

package main

import (
	"crypto/sha256"
	"encoding/binary"
	"fmt"
	"sort"
	"strconv"
)

type Ring struct {
	points []uint32
	owner  map[uint32]string
}

// hash needs good mixing: names like "a#0", "a#1" differ in one byte and must land far apart.
func hash(s string) uint32 {
	sum := sha256.Sum256([]byte(s))
	return binary.BigEndian.Uint32(sum[:4])
}

func NewRing(nodes []string, replicas int) *Ring {
	r := &Ring{owner: map[uint32]string{}}
	for _, n := range nodes {
		for i := 0; i < replicas; i++ {
			p := hash(n + "#" + strconv.Itoa(i))
			r.points = append(r.points, p)
			r.owner[p] = n
		}
	}
	sort.Slice(r.points, func(i, j int) bool { return r.points[i] < r.points[j] })
	return r
}

func (r *Ring) Get(key string) string {
	h := hash(key)
	i := sort.Search(len(r.points), func(i int) bool { return r.points[i] >= h })
	if i == len(r.points) {
		i = 0 // wrap around the circle
	}
	return r.owner[r.points[i]]
}

func main() {
	const keys = 10000
	nodes := []string{"a", "b", "c"}
	ring := NewRing(nodes, 100)
	before := make([]string, keys)
	counts := map[string]int{}
	for i := range before {
		before[i] = ring.Get("user-" + strconv.Itoa(i))
		counts[before[i]]++
	}
	fmt.Println("keys per node with 100 virtual nodes:", counts)

	smaller := NewRing([]string{"a", "b"}, 100)
	movedRing, movedMod := 0, 0
	for i := 0; i < keys; i++ {
		k := "user-" + strconv.Itoa(i)
		if smaller.Get(k) != before[i] {
			movedRing++
		}
		if int(hash(k))%3 != int(hash(k))%2 { // modulo placement before and after losing one of three nodes
			movedMod++
		}
	}
	fmt.Printf("remove node c: ring moved %d of %d keys, modulo moved %d\n", movedRing, keys, movedMod)

	few := NewRing(nodes, 1)
	fewCounts := map[string]int{}
	for i := 0; i < keys; i++ {
		fewCounts[few.Get("user-"+strconv.Itoa(i))]++
	}
	fmt.Println("keys per node with 1 virtual node:   ", fewCounts)
}

keys per node with 100 virtual nodes: map[a:3814 b:2753 c:3433]
remove node c: ring moved 3433 of 10000 keys, modulo moved 6651
keys per node with 1 virtual node:    map[a:5519 b:4112 c:369]

Two things in that output are worth reading carefully. First, removing node c moved 3,433 keys, which is exactly the number that lived on c, while modulo placement moved 6,651 of 10,000. That is the property you buy a ring for. Second, even with 100 virtual nodes the split is 3,814, 2,753 and 3,433, a spread of about plus or minus 15 percent around the ideal 3,333, and with a single point per node it is 5,519, 4,112 and 369. More virtual nodes tighten the spread; a few hundred per node is common.

My first version of this program used hash/fnv, since it is fast and in the standard library, and the split with 100 points came out as 2,742, 5,830 and 1,428. FNV-1a mixes poorly when inputs differ in a single trailing byte, which is precisely what a#0, a#1 and a#2 do, so a node's points clumped together on the circle. The hash function above takes the first four bytes of a SHA-256 instead. It is slower, but it runs once per request on a short string, and an uneven ring is a worse problem than a few hundred nanoseconds. Whatever hash you use, measure the distribution as this program does before trusting it.

To add stickiness to the balancer, replace pick with ring.Get(key) for requests that carry a routing key (a session cookie, a tenant header), keep a ring per set of live backends, and rebuild it when health changes. If the chosen node is down, walk to the next point on the ring rather than falling back to a random backend, so the key keeps a stable second home.

Stickiness is a trade. Keys are never perfectly even, a single hot tenant can overload its node, and state kept in memory dies with that node. Hash routing is a cache optimisation. Anything that must survive a backend failing belongs in a shared store.

What to measure on a balancer

The pool tracks inflight already, so exporting it costs nothing. The numbers I would want on a dashboard are requests per backend (a skew means the algorithm or the keys are uneven), in-flight per backend (a climb means a backend is slowing), the count of 502 and 503 responses (passive failures and empty pools), and the time each backend spent marked down. Log every state flip with the reason, probe failure or proxy error, because "why was B out of rotation at 3 a.m." is otherwise unanswerable. Metrics with expvar is enough to start, and a Prometheus client when you outgrow it.

Draining a backend on purpose

Removing a backend for a deploy is different from a crash, because you want its in-flight requests to finish. The pool already has what is needed: stop picking the backend (set alive to false, or add a separate draining flag so the checker does not undo it), then wait until inflight reaches zero or a deadline passes, then stop the process. The wait is a loop with a ticker, ten lines, and it is the balancer-side twin of graceful shutdown on the server. Getting both ends right is what makes a rolling deploy invisible.

Choosing the algorithm

Round robin gives each backend an equal share of requests, and it is what I would deploy unless measurements say otherwise. It assumes requests cost about the same and the backends are about equally fast.

Least connections compares in-flight requests. It helps when durations vary widely (a mix of 5 ms lookups and 5 second reports) because a backend stuck on slow work stops receiving more. For uniform fast requests it behaves like round robin with extra bookkeeping.

Weighted variants exist for mixed hardware: give the bigger machine a weight and pick proportionally. Consistent hashing sends the same key (a user, a tenant, a cache key) to the same backend, which matters for caches and for state kept in memory; it is a different tool, and the usual Go building block is a hash ring with virtual nodes.

Where this stops being enough

This balancer is a good fit for a handful of backends that you control, an internal tool, or a development environment. It does not do TLS to backends with certificate verification options, slow start for backends that just came back, connection draining when you remove a backend deliberately, or global limits. Kubernetes Service objects, nginx's upstream module, HAProxy and cloud load balancers have all of those, and in production you should usually use one. The value of writing this is that you now know exactly what they are doing, and you can read their health check settings (interval, rise, fall) without guessing.

To go further, build an API gateway on top, adding authentication and per-route policy in front of the pool.

Testing a balancer

Most of this code can be tested without a network. pick is a function from state to a backend, so table-test it with hand-set alive and inflight values, as the last lines of the demo do. The health thresholds are tested by calling Check against fake servers and asserting on alive after each call, which is what the walk through above was, and no sleeps are needed because Check is synchronous. The passive path is tested by closing a server and sending one request.

Two tests are easy to forget. The first runs many goroutines through ServeHTTP while another flips health state, under -race, because the whole design rests on those atomics being the only shared state. The second checks the all-down case returns 503 with Retry-After and not a panic on a nil backend; it is the line most likely to be broken by a refactor, and it only matters during an outage, which is exactly when you cannot afford the bug.