gRPC Retries Hedging and Service Config in Go Clients

Networks fail in small ways all the time: a connection reset, a server restarting during a deployment, a pod that vanished a second ago. For most of these, trying again works. gRPC can do the retrying for you, transparently, inside the client, with a policy you set in configuration and no code at the call site. That is attractive and slightly dangerous, because retries are also the way a small failure becomes a large outage.

This article shows how to configure retries in grpc-go, what the policy knobs do, how throttling limits the damage, and what to do about hedging, which the gRPC specification describes and the Go library does not implement. It continues from error codes, which decide what is retryable, and deadlines, which bound the total effort, and ties into the general rules in timeouts and retries across services. It ran on Go 1.25.6.

The service config

grpc-go reads a JSON service config that describes per-method behaviour: timeouts, retry policy, load balancing. The client can take it from the name resolver (DNS TXT records, a control plane) or from a default you give at dial time with grpc.WithDefaultServiceConfig. The lab uses the default form.

syntax = "proto3";
package flaky.v1;
option go_package = "example.test/lab/flakypb";

service Flaky {
  rpc Do(DoRequest) returns (DoReply);
}

// fail_first says how many attempts of this call id should fail with the given code before one succeeds.
message DoRequest { string call_id = 1; int32 fail_first = 2; string code = 3; int32 delay_ms = 4; }
message DoReply { int32 attempt = 1; }

The retry policy:

{
  "methodConfig": [{
    "name": [{"service": "flaky.v1.Flaky"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.01s",
      "maxBackoff": "0.05s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

name selects methods: a service (all its methods), a service plus method, or an empty object for all methods. maxAttempts counts the first attempt, so 4 means up to three retries. The backoff fields give an exponential delay starting at initialBackoff, multiplied by backoffMultiplier each time and capped at maxBackoff; grpc-go applies random jitter. retryableStatusCodes lists the codes (as names) that cause a retry. The short values in the lab keep the demo fast; production values are in the hundreds of milliseconds and seconds.

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sync"
	"sync/atomic"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/status"

	pb "example.test/lab/flakypb"
)

type flaky struct {
	pb.UnimplementedFlakyServer
	mu       sync.Mutex
	attempts map[string]int
	total    atomic.Int64
}

func (f *flaky) Do(ctx context.Context, r *pb.DoRequest) (*pb.DoReply, error) {
	f.total.Add(1)
	f.mu.Lock()
	f.attempts[r.GetCallId()]++
	n := f.attempts[r.GetCallId()]
	f.mu.Unlock()
	if r.GetDelayMs() > 0 && n == 1 {
		select {
		case <-time.After(time.Duration(r.GetDelayMs()) * time.Millisecond):
		case <-ctx.Done():
			return nil, ctx.Err()
		}
	}
	if int32(n) <= r.GetFailFirst() {
		code := codes.Unavailable
		if r.GetCode() == "invalid" {
			code = codes.InvalidArgument
		}
		return nil, status.Error(code, fmt.Sprintf("attempt %d fails", n))
	}
	return &pb.DoReply{Attempt: int32(n)}, nil
}

const retryConfig = `{
  "methodConfig": [{
    "name": [{"service": "flaky.v1.Flaky"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.01s",
      "maxBackoff": "0.05s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}`

const throttledConfig = `{
  "methodConfig": [{
    "name": [{"service": "flaky.v1.Flaky"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.001s",
      "maxBackoff": "0.005s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }],
  "retryThrottling": {"maxTokens": 4, "tokenRatio": 0.1}
}`

func main() {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	impl := &flaky{attempts: map[string]int{}}
	srv := grpc.NewServer()
	pb.RegisterFlakyServer(srv, impl)
	go srv.Serve(lis)
	defer srv.Stop()

	client := func(opts ...grpc.DialOption) pb.FlakyClient {
		conn, err := grpc.NewClient(lis.Addr().String(), append([]grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}, opts...)...)
		if err != nil {
			log.Fatal(err)
		}
		return pb.NewFlakyClient(conn)
	}
	call := func(c pb.FlakyClient, id string, failFirst int32, code string) (string, int64) {
		before := impl.total.Load()
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		rep, err := c.Do(ctx, &pb.DoRequest{CallId: id, FailFirst: failFirst, Code: code})
		made := impl.total.Load() - before
		if err != nil {
			return status.Code(err).String(), made
		}
		return fmt.Sprintf("OK on attempt %d", rep.GetAttempt()), made
	}

	withRetry := client(grpc.WithDefaultServiceConfig(retryConfig))
	noRetry := client()
	for _, tc := range []struct {
		label string
		c     pb.FlakyClient
		fail  int32
		code  string
	}{
		{"retry policy, 2 failures then success", withRetry, 2, ""},
		{"retry policy, fails more than 4 times", withRetry, 9, ""},
		{"retry policy, InvalidArgument is not retryable", withRetry, 9, "invalid"},
		{"no retry policy, 2 failures", noRetry, 2, ""},
	} {
		res, made := call(tc.c, tc.label, tc.fail, tc.code)
		fmt.Printf("%-50s -> %-18s requests that reached the server: %d\n", tc.label, res, made)
	}

	// Retry throttling: when many calls fail, retries stop being sent so a struggling server is not hit 4 times as hard.
	for _, cfg := range []struct {
		label string
		c     pb.FlakyClient
	}{{"without throttling", client(grpc.WithDefaultServiceConfig(retryConfig))}, {"with throttling", client(grpc.WithDefaultServiceConfig(throttledConfig))}} {
		before := impl.total.Load()
		for i := 0; i < 20; i++ {
			call(cfg.c, fmt.Sprintf("%s-%d", cfg.label, i), 99, "")
		}
		fmt.Printf("20 calls that always fail, %-18s: %d requests reached the server (4 per call would be 80)\n", cfg.label, impl.total.Load()-before)
	}

	// Hedging is in the service config spec, but grpc-go does not implement it, so do it in application code.
	hedge := func(delay time.Duration, call func(ctx context.Context) (string, error)) (string, time.Duration) {
		start := time.Now()
		ctx, cancel := context.WithCancel(context.Background())
		defer cancel()
		results := make(chan string, 2)
		launch := func() {
			if v, err := call(ctx); err == nil {
				results <- v
			}
		}
		go launch()
		select {
		case v := <-results:
			return v, time.Since(start)
		case <-time.After(delay):
		}
		go launch() // the first attempt is slow: send a second and take whichever answers first
		return <-results, time.Since(start)
	}
	v, took := hedge(50*time.Millisecond, func(ctx context.Context) (string, error) {
		rep, err := withRetry.Do(ctx, &pb.DoRequest{CallId: "hedge", DelayMs: 400})
		if err != nil {
			return "", err
		}
		return fmt.Sprintf("answered by attempt %d", rep.GetAttempt()), nil
	})
	fmt.Printf("hedged call (first attempt stalls 400ms, hedge after 50ms): %s, took under 200ms: %v\n", v, took < 200*time.Millisecond)
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

retry policy, 2 failures then success              -> OK on attempt 3    requests that reached the server: 3
retry policy, fails more than 4 times              -> Unavailable        requests that reached the server: 4
retry policy, InvalidArgument is not retryable     -> InvalidArgument    requests that reached the server: 1
no retry policy, 2 failures                        -> Unavailable        requests that reached the server: 1
20 calls that always fail, without throttling: 80 requests reached the server (4 per call would be 80)
20 calls that always fail, with throttling   : 21 requests reached the server (4 per call would be 80)
hedged call (first attempt stalls 400ms, hedge after 50ms): answered by attempt 2, took under 200ms: true

What the run shows

A transient failure is hidden from the caller. The first call failed twice with Unavailable and succeeded on attempt 3. The code that made the call saw one successful response. The server saw three requests.

Retries are bounded. The second call always failed. It stopped at four requests and the caller got Unavailable, the status of the last attempt.

Only listed codes retry. The third call failed with InvalidArgument, which is not in retryableStatusCodes. One request, no retry. This is the correct behaviour: a malformed request will not become valid on the second try. The fourth call used a client with no policy and a failure that would have been retried: also one request. There are no retries by default.

The code choice is the whole safety story. UNAVAILABLE means the request probably did not run, or the server is telling you to come back. Adding DEADLINE_EXCEEDED, ABORTED or INTERNAL risks repeating an operation that may have completed. Only retry methods that are idempotent, or that carry an idempotency key.

What gRPC does without being asked

Even with a policy, there are two cases grpc-go retries with no configuration. Transparent retry happens when a request never left the client, for example because the connection was lost before the headers were written: the library tries again on another connection, since nothing could have run. And a call that fails with a closed connection after the server received it is not retried unless the status is in your list, because the server may have processed it.

Retry throttling

Retries add load. A server at its limit starts returning Unavailable, every client retries three more times, the server's load quadruples and it never recovers. The defence is retry throttling, configured beside the retry policy:

"retryThrottling": {"maxTokens": 4, "tokenRatio": 0.1}

Each client connection keeps a token count that starts at maxTokens. A failed attempt removes one token; a success adds tokenRatio. When the count is at or below half of maxTokens, retries are not sent; calls get only their first attempt. Under normal conditions the bucket stays full and retries happen. Under sustained failure it drains and the retries stop.

The lab measures it. Twenty calls that always fail, with a retry policy and no throttling: 80 requests reached the server, four per call. With throttling: 21 requests. The first call used up the bucket (one retry), after which each call made only its first attempt. Retry load dropped by three quarters at the moment it would have hurt most. In production use larger bucket sizes (the example uses 4 so the effect shows in 20 calls), but the shape of the curve is the same.

Hedging, and why you have to write it

The gRPC service config also defines a hedgingPolicy: send the same request to multiple backends, staggered, and take the first answer. It attacks tail latency: if the 99th percentile is slow because a request sometimes lands on a slow instance, a second request after the median delay usually avoids it. The cost is extra load and the need for idempotent methods.

grpc-go does not implement hedging. The source contains a TODO(hedging) comment in the stream code and the policy is not applied. Do not configure it and assume it works; a config that has no effect is worse than none. If you want hedging, do it in application code. The lab's hedge function starts a call, waits 50 ms, and if there is no answer starts a second one, returning whichever succeeds first and cancelling the rest through the shared context. With the first attempt stalled for 400 ms, the hedged call returned from the second attempt in well under 200 ms.

Rules for hedging by hand:

• Only for idempotent reads, or writes with idempotency keys.
• Delay the second request until about the p95 latency. A shorter delay doubles your load; a longer one saves nothing.
• Cap hedges per call (one is usually enough) and budget them overall (no more than a few percent of requests), so a slow period does not double the traffic.
• Cancel the loser as soon as there is a winner.
• Count hedged requests in metrics so you can see the cost.

Retries and deadlines together

The per-call deadline covers all attempts, so retries share one budget. A call with a 1 second deadline and a first attempt that fails at 900 ms has 100 ms for the retry. Set the deadline from the user's point of view, set maxAttempts modestly, and check that maxAttempts * attempt time + backoff can fit. If it cannot, the later attempts never run, and you have configured behaviour you do not get.

Retries in layers

The biggest mistake is retrying at several levels. If the client library retries 3 times, and the service behind it retries its downstream 3 times, and that retries its database 3 times, one failure at the bottom turns into 27 attempts. Retry at one layer, usually closest to the original caller, or use a budget. Timeouts, retries and deadlines across calls measures this multiplication.

Observability and testing

Log or count retries, or you will not know they are happening: grpc-go exposes them through stats handlers and the grpc.retries metrics in OpenTelemetry instrumentation (gRPC and OpenTelemetry). To test, use a scripted server like the one in the lab that fails a known number of times per call id, and assert on the attempt count and final status. Test both the success after N failures and the exhaustion case. The retry design and the exact semantics are defined in the gRPC retry guide and the service config format in the gRPC documentation.