gRPC Load Balancing in Go Clients and Servers Explained

HTTP/1.1 load balancing is easy to reason about. Each request opens, or reuses, a connection; a balancer in front sends each request to a backend; with thousands of requests from thousands of clients the load evens out. gRPC breaks that picture, and the result is a common production surprise: you deploy ten replicas of a service, put them behind a load balancer, and one of them is at 90 percent CPU while nine are idle.

The cause is in the transport. This article explains why, shows the effect with a runnable experiment, and covers the practical ways to balance gRPC: client-side policies, headless services, proxies and the connection-recycling settings that make any of them work. It builds on the gRPC tutorial and assumes you have seen retries. It ran on Go 1.25.6.

Why gRPC load balancing is different

A gRPC client opens an HTTP/2 connection to the server and keeps it open. Every call is a stream multiplexed over that connection, and thousands of calls can share it. There is no new connection per call and nothing for a connection-level balancer to rebalance.

A classic layer-4 load balancer (a cloud network load balancer, kube-proxy with a normal Kubernetes ClusterIP service, HAProxy in TCP mode) picks a backend when a TCP connection opens and then forwards bytes. With HTTP/1.1 and short connections, that approximates per-request balancing. With gRPC, one client's connection lands on one backend and stays there. If a client makes all its calls over one connection, one backend receives all of that client's load. With many clients the load evens out statistically, but with a few heavy clients (a gateway, a batch job) it does not, and when you add replicas, existing connections do not move, so the new backends sit idle until clients reconnect.

There are three families of fixes:

1. Client-side load balancing. The client knows all the backend addresses and spreads calls itself.
2. A layer-7 proxy that understands HTTP/2. Envoy, NGINX (with grpc_pass), Linkerd or a cloud application load balancer terminates the client connection and sends each call, as a stream, to a backend over its own connections.
3. Lookaside balancing, where a control plane (xDS) tells clients where to send calls. This is what service meshes and Google's infrastructure use.

The experiment

The lab starts four backends, A to D, each of which answers "I am X" to a Name call. A client uses a manual resolver that lets the program decide which addresses the client sees, which is the same mechanism a real resolver (DNS, a service registry, xDS) uses.

syntax = "proto3";
package whoami.v1;
option go_package = "example.test/lab/whoamipb";

service WhoAmI {
  rpc Name(NameRequest) returns (NameReply);
}

message NameRequest {}
message NameReply { string backend = 1; }

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sort"
	"strings"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/resolver"
	"google.golang.org/grpc/resolver/manual"

	pb "example.test/lab/whoamipb"
)

type backend struct {
	pb.UnimplementedWhoAmIServer
	name string
}

func (b backend) Name(context.Context, *pb.NameRequest) (*pb.NameReply, error) {
	return &pb.NameReply{Backend: b.name}, nil
}

func startBackend(name string) (addr string, stop func()) {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	srv := grpc.NewServer()
	pb.RegisterWhoAmIServer(srv, backend{name: name})
	go srv.Serve(lis)
	return lis.Addr().String(), srv.Stop
}

func spread(counts map[string]int) string {
	var parts []string
	for k, v := range counts {
		parts = append(parts, fmt.Sprintf("%s=%d", k, v))
	}
	sort.Strings(parts)
	return strings.Join(parts, " ")
}

func calls(c pb.WhoAmIClient, n int) (map[string]int, int) {
	counts := map[string]int{}
	failed := 0
	for i := 0; i < n; i++ {
		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
		r, err := c.Name(ctx, &pb.NameRequest{})
		cancel()
		if err != nil {
			failed++
			continue
		}
		counts[r.GetBackend()]++
	}
	return counts, failed
}

func main() {
	addrs := map[string]string{}
	stops := map[string]func(){}
	for _, n := range []string{"A", "B", "C", "D"} {
		addrs[n], stops[n] = startBackend(n)
	}
	defer func() {
		for _, s := range stops {
			s()
		}
	}()
	byAddr := func(names ...string) []resolver.Address {
		var out []resolver.Address
		for _, n := range names {
			out = append(out, resolver.Address{Addr: addrs[n]})
		}
		return out
	}

	dial := func(r *manual.Resolver, config string) pb.WhoAmIClient {
		opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithResolvers(r)}
		if config != "" {
			opts = append(opts, grpc.WithDefaultServiceConfig(config))
		}
		conn, err := grpc.NewClient(r.Scheme()+":///whoami", opts...)
		if err != nil {
			log.Fatal(err)
		}
		return pb.NewWhoAmIClient(conn)
	}
	warm := func(c pb.WhoAmIClient) {
		for i := 0; i < 8; i++ { // let every subchannel connect before measuring
			ctx, cancel := context.WithTimeout(context.Background(), time.Second)
			c.Name(ctx, &pb.NameRequest{}, grpc.WaitForReady(true))
			cancel()
			time.Sleep(10 * time.Millisecond)
		}
	}

	// 1. The default policy, pick_first: connect to the first address that works and send everything there.
	r1 := manual.NewBuilderWithScheme("pf")
	r1.InitialState(resolver.State{Addresses: byAddr("A", "B", "C")})
	c1 := dial(r1, "")
	counts, _ := calls(c1, 30)
	fmt.Println("pick_first, 3 backends, 30 calls:   ", spread(counts))

	// 2. round_robin: one connection per backend, calls rotate across them.
	const rr = `{"loadBalancingConfig":[{"round_robin":{}}]}`
	r2 := manual.NewBuilderWithScheme("rr")
	r2.InitialState(resolver.State{Addresses: byAddr("A", "B", "C")})
	c2 := dial(r2, rr)
	warm(c2)
	counts, _ = calls(c2, 30)
	fmt.Println("round_robin, 3 backends, 30 calls:  ", spread(counts))

	// 3. The resolver learns about a new backend (a scale-up). The client rebalances without a restart.
	r2.UpdateState(resolver.State{Addresses: byAddr("A", "B", "C", "D")})
	time.Sleep(100 * time.Millisecond)
	warm(c2)
	counts, _ = calls(c2, 40)
	fmt.Println("after the resolver adds D, 40 calls:", spread(counts))

	// 4. A backend dies without telling the resolver. Retries hide the gap while the client notices.
	const rrRetry = `{"loadBalancingConfig":[{"round_robin":{}}],
	  "methodConfig":[{"name":[{"service":"whoami.v1.WhoAmI"}],"retryPolicy":{"maxAttempts":3,"initialBackoff":"0.01s","maxBackoff":"0.05s","backoffMultiplier":2,"retryableStatusCodes":["UNAVAILABLE"]}}]}`
	r3 := manual.NewBuilderWithScheme("rrr")
	r3.InitialState(resolver.State{Addresses: byAddr("A", "B", "C", "D")})
	c3 := dial(r3, rrRetry)
	warm(c3)
	stops["C"]()
	counts, failed := calls(c3, 30)
	fmt.Printf("backend C stopped, 30 calls with retries: %s, failed=%d\n", spread(counts), failed)
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

pick_first, 3 backends, 30 calls:    A=30
round_robin, 3 backends, 30 calls:   A=10 B=10 C=10
after the resolver adds D, 40 calls: A=10 B=10 C=10 D=10
backend C stopped, 30 calls with retries: A=9 B=10 D=11, failed=0

pick_first: the default

The client is given three addresses. grpc-go's default policy, pick_first, connects to the first address that works and uses only that one. It keeps the others as fallbacks. The first line of output: 30 calls, A=30. All of them went to A. This is correct for a single service address and wrong when the address list is a set of equivalent replicas.

round_robin: one connection per backend

Setting {"loadBalancingConfig":[{"round_robin":{}}]} as the default service config changes the policy. The client now opens a connection to every address it was given, and each call goes to the next connection in turn. The second line: A=10 B=10 C=10. The load is even.

Note the warm-up in the lab: the client connects lazily and in the background, so the first few calls can be sent before every subchannel is ready. A production client does not notice; a benchmark or test must wait or it measures start-up.

Resolver updates

The program then calls UpdateState on the manual resolver with four addresses, as a scale-up would. The client reconciles: it opens a connection to D, and the next 40 calls split A=10 B=10 C=10 D=10. No restart, no reconnect of the existing three. In a real deployment, the DNS resolver or service registry provides the updates. The important property is that the list changes over time and the client follows it.

A backend dies

The last scenario stops backend C without telling the resolver (a crash, an eviction). The connection to C breaks, its subchannel goes into a failure state and the round-robin picker drops it, but there is a window before that happens in which calls routed to C fail with Unavailable. The client in this scenario has a retry policy for UNAVAILABLE, and all 30 calls succeeded, spread over A, B and D. The distribution (9, 10 and 11 in the recorded run) is approximately even and not exact, because the picker rotation continued while the set changed. Without the retry policy, a few calls would have failed during the window; with it, the failure is invisible to the caller. That is the reason client-side balancing and retries belong together.

Resolvers: where the address list comes from

The channel target has a scheme: dns:///my-service:50051, passthrough:///host:port, unix:///path, or a custom one. The resolver for that scheme produces the address list.

grpc.NewClient defaults to dns. The older grpc.Dial defaulted to passthrough, which uses the target string as the address without resolution. With NewClient, a target like localhost:50051 goes through the DNS resolver. That is usually what you want, and occasionally a surprise when a name resolves to several addresses. If you need the old behaviour, write passthrough:///localhost:50051 explicitly. (The lab uses the manual resolver, so neither applies.)

The DNS resolver re-resolves rarely. It resolves at start-up and again when a connection fails, with a minimum interval between lookups, but it does not poll on a timer. After a scale-up, a client holding live connections to the old replicas may not learn about the new ones for a long time. Combine DNS with server-side connection recycling (below) to force clients to reconnect and re-resolve.

Kubernetes. A normal ClusterIP Service gives clients one virtual IP; kube-proxy balances connections, which is the problem described above. A headless Service (clusterIP: None) makes DNS return the pod addresses, and dns:///my-svc.my-namespace.svc.cluster.local:50051 with round_robin then spreads calls across pods:

apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  clusterIP: None
  selector:
    app: orders
  ports:
    - port: 50051

Pods that start or stop change the DNS answer; the client picks it up when it next resolves. For faster reaction, use a service mesh or an xDS control plane, or a resolver that watches the Kubernetes API.

Server-side settings that make balancing work

Client-side balancing assumes clients reconnect and re-resolve from time to time. Make that happen from the server:

• MaxConnectionAge (keepalive.ServerParameters) closes a connection after a time, with a MaxConnectionAgeGrace period for in-flight calls to finish. A client sees the connection end gracefully (the server sends GOAWAY) and reconnects, which triggers re-resolution. Set it to minutes, with random jitter (grpc-go adds jitter), so connections do not all recycle at once (keepalive).
• MaxConnectionIdle closes connections that carry no calls, freeing resources.
• Graceful shutdown (GracefulStop) sends GOAWAY to all clients so they move to other backends cleanly (graceful shutdown).

Proxies

If you cannot change the clients, put an L7 proxy in front. Envoy is the common choice: it terminates the client's HTTP/2 connection, keeps a pool of connections to the backends, and picks a backend per call with round-robin, least-request or consistent-hash policies, plus health checks and outlier detection. NGINX supports grpc_pass for proxying gRPC with per-call balancing. Cloud application load balancers (such as AWS ALB with gRPC support, or Google's HTTP(S) load balancers) do the same as a managed service. The trade-off is an extra hop and a component to operate, in exchange for balancing that clients cannot get wrong. I did not run a proxy in this article's lab.

A third option for sidecar-style deployments is a service mesh such as Linkerd or Istio, where a sidecar proxy beside each client does per-call balancing without the application knowing.

Choosing a policy

pick_first for talking to one endpoint: a single server, or an address that is itself a balanced proxy.

round_robin for a set of equivalent replicas when you have a list of addresses. It is simple and works well for uniform requests.

Weighted round robin, available in grpc-go as weighted_round_robin, uses load reports from the backends (ORCA) to prefer less loaded ones. It needs backend support and helps when request costs vary a lot.

Consistent hashing (ring hash), available through xDS, keeps requests with the same key on the same backend, useful for caches. It is not available in plain client-side config.

Least-request balancing is common in Envoy and appears in xDS-driven clients; plain grpc-go gives you the policies above.

Health, outliers and subsetting

The round-robin picker only knows about connectivity. A backend that accepts connections and answers every call with errors still receives its share. Two mechanisms help: client-side health checking, where the client calls the standard health service (health checking) and removes unhealthy backends from rotation, and outlier detection in a proxy or mesh, which ejects backends with high error rates.

In very large deployments, every client connecting to every backend produces a huge number of connections. Subsetting gives each client a stable random subset of backends. That is an operational optimisation for hundreds of backends and not something to do early.

A practical rollout recipe

If you run on Kubernetes and own both the clients and the servers, this sequence works and can be adopted one step at a time.

1. Measure first. Export per-pod request counts for the server (grpc_server_handled_total or your own counter) and look at the spread. If it is even already, your clients are many and numerous and you may not need to change anything.
2. Make the Service headless and switch clients to dns:///name.namespace.svc.cluster.local:port with round_robin. Do it for one client first and watch the spread shift.
3. Add retries on Unavailable for idempotent methods with a throttling policy, so pod restarts are invisible (retries).
4. Set MaxConnectionAge on servers, for example 5 minutes with a 30 second grace, so clients re-resolve and pick up new pods after a scale-up. Confirm in staging that a new replica receives traffic within about that time.
5. Handle shutdown properly: GracefulStop on SIGTERM, and a preStop hook or readiness change so the pod is removed from DNS before it stops accepting calls.
6. Consider a mesh or proxy if you have many languages and teams, and each team would otherwise implement the above on their own.

Mistakes that look like balancing problems

One shared connection across goroutines is fine; a new connection per request is not. A common mistake in the other direction is to create a grpc.ClientConn per call so that a layer-4 balancer spreads the connections. It does spread them, at the cost of a TCP and TLS handshake per call, which is slower and wastes server resources. Keep one ClientConn per target for the life of the process, and balance calls inside it.

A single ClusterIP address with round_robin. The client has one address, so round-robin has one connection to rotate over. Nothing is balanced. The policy only helps when the resolver returns several addresses.

Stale DNS caches. The OS, a sidecar or a corporate resolver caching an answer for minutes will defeat quick scaling. Check the TTL of the records and the resolver's behaviour.

Hot keys. Round-robin balances calls, not work. If one call in a hundred does a thousand times more work, backends will still be uneven. Look at latency and CPU per backend, not only counts, and consider least-request balancing in a proxy.

Streams are balanced once. A long-lived stream stays on the backend it started on. Many long streams and a deployment that adds backends leave the new ones idle until streams end. Have servers end streams after a maximum age, and make clients reconnect (streaming).

Ignoring TLS. When a mesh or proxy terminates TLS, the backend sees the proxy's connection and not the client's identity. Pass identity in metadata or use mutual TLS between layers (authentication).

Testing

You can test balancing with in-process servers and a manual resolver exactly as the lab does. Assert on counts with tolerance, since rotation depends on start-up timing, and test the failure path: stop a backend, make calls, assert that none fail with retries on and that the dead backend gets no traffic after the picker updates. Test resolver updates in both directions. In staging, verify the production recipe: that a deployment of a new version moves traffic, and that scaling up actually gives new replicas their share within the time you expect. The gRPC project documents the options in its load balancing blog post and guide, and the policy configuration in the service config documentation.