gRPC Keepalive Settings That Prevent Silent Disconnects

TCP does not tell you when the other side disappears. If a process crashes, the operating system closes its sockets and sends a FIN or a RST, and you find out at once. If a laptop closes its lid, a NAT gateway forgets the mapping, a cable is pulled or a firewall silently drops packets, nothing is sent. The connection looks open to both ends until one of them tries to write and waits through retransmission timeouts, which can take many minutes. A gRPC stream waiting to read is not writing, so it can wait for hours, holding a goroutine, memory and perhaps a database cursor.

Keepalive fixes this by making an idle connection prove it is alive. This article shows it working, with a network path that is killed silently during a stream, explains the client, server and enforcement settings, and covers the one configuration mistake that produces a baffling error. It builds on server streaming and relates to load balancing and graceful shutdown. It ran on Go 1.25.6.

What keepalive does

gRPC runs over HTTP/2, which has a PING frame: a small message that the receiver must answer promptly. Keepalive sends a ping when a connection has had no activity for a time Time. If no reply (or any other frame) arrives within Timeout, the connection is declared dead and closed, which cancels the contexts of the calls on it. A live peer answers within milliseconds, so the cost is a few bytes every so often.

This is different from TCP keepalive, a socket option that works at the kernel level with defaults of two hours. Application-level HTTP/2 pings are faster to configure, work through proxies that terminate TCP, and detect a wedged process that is still holding the socket open.

The experiment

The lab places a small TCP forwarder between a client and a server. It can be switched into a mode where it reads bytes and drops them: no close, no reset, exactly what a dead network path looks like. The client opens a streaming call (Hold) and the server's handler blocks on the stream's context. Then the forwarder goes dead. The server has keepalive with Time: 1s and Timeout: 1s.

syntax = "proto3";
package watch.v1;
option go_package = "example.test/lab/watchpb";

service Watcher {
  // Hold blocks until the client goes away and tells the server how it noticed.
  rpc Hold(HoldRequest) returns (stream HoldEvent);
  rpc Ping(HoldRequest) returns (HoldEvent);
}

message HoldRequest {}
message HoldEvent { string note = 1; }

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sync/atomic"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/connectivity"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/keepalive"

	pb "example.test/lab/watchpb"
)

type watcher struct {
	pb.UnimplementedWatcherServer
	gone chan time.Time
}

func (w *watcher) Hold(_ *pb.HoldRequest, s grpc.ServerStreamingServer[pb.HoldEvent]) error {
	s.Send(&pb.HoldEvent{Note: "holding"})
	<-s.Context().Done() // returns when the peer cancels, disconnects or is declared dead
	w.gone <- time.Now()
	return s.Context().Err()
}

func (w *watcher) Ping(context.Context, *pb.HoldRequest) (*pb.HoldEvent, error) {
	return &pb.HoldEvent{Note: "pong"}, nil
}

// blackhole is a TCP forwarder that can silently stop moving bytes: no FIN, no RST, like a dead NAT entry or a pulled cable.
type blackhole struct {
	lis  net.Listener
	dead atomic.Bool
}

func newBlackhole(target string) *blackhole {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	b := &blackhole{lis: lis}
	go func() {
		for {
			c, err := lis.Accept()
			if err != nil {
				return
			}
			up, err := net.Dial("tcp", target)
			if err != nil {
				c.Close()
				continue
			}
			go b.pipe(up, c)
			go b.pipe(c, up)
		}
	}()
	return b
}

func (b *blackhole) pipe(dst, src net.Conn) {
	buf := make([]byte, 32*1024)
	for {
		n, err := src.Read(buf)
		if n > 0 && !b.dead.Load() { // when dead, bytes are read and thrown away
			dst.Write(buf[:n])
		}
		if err != nil {
			return
		}
	}
}

func main() {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	w := &watcher{gone: make(chan time.Time, 1)}
	// The server pings a connection that has been quiet for 1s and gives it 1s to answer.
	srv := grpc.NewServer(grpc.KeepaliveParams(keepalive.ServerParameters{Time: time.Second, Timeout: time.Second}))
	pb.RegisterWatcherServer(srv, w)
	go srv.Serve(lis)
	defer srv.Stop()

	proxy := newBlackhole(lis.Addr().String())
	conn, err := grpc.NewClient(proxy.lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	c := pb.NewWatcherClient(conn)

	stream, err := c.Hold(context.Background(), &pb.HoldRequest{})
	if err != nil {
		log.Fatal(err)
	}
	ev, _ := stream.Recv()
	fmt.Println("stream open, server said:", ev.GetNote())

	start := time.Now()
	proxy.dead.Store(true) // the network path silently dies; nobody sends a FIN or a RST
	select {
	case at := <-w.gone:
		d := at.Sub(start)
		fmt.Printf("server declared the peer dead after %.0f seconds (1s idle + 1s ping timeout, rounded): %v\n", d.Seconds(), d > time.Second && d < 4*time.Second)
	case <-time.After(10 * time.Second):
		fmt.Println("server never noticed within 10 seconds")
	}

	// Without keepalive the same silence would last until the operating system gave up (minutes to hours).
	// Server-side idle and age limits recycle connections even when they are healthy.
	lis2, _ := net.Listen("tcp", "127.0.0.1:0")
	srv2 := grpc.NewServer(grpc.KeepaliveParams(keepalive.ServerParameters{MaxConnectionIdle: 300 * time.Millisecond}))
	pb.RegisterWatcherServer(srv2, &watcher{gone: make(chan time.Time, 1)})
	go srv2.Serve(lis2)
	defer srv2.Stop()
	conn2, _ := grpc.NewClient(lis2.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
	defer conn2.Close()
	ctx := context.Background()
	pb.NewWatcherClient(conn2).Ping(ctx, &pb.HoldRequest{})
	fmt.Println("after a call, connection state:", conn2.GetState())
	time.Sleep(900 * time.Millisecond)
	conn2.WaitForStateChange(ctx, connectivity.Ready)
	fmt.Println("after 0.9s idle (MaxConnectionIdle 300ms), state:", conn2.GetState())
	_, err = pb.NewWatcherClient(conn2).Ping(ctx, &pb.HoldRequest{})
	fmt.Println("next call reconnects transparently, error:", err, "- state now:", conn2.GetState())
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

stream open, server said: holding
server declared the peer dead after 2 seconds (1s idle + 1s ping timeout, rounded): true
after a call, connection state: READY
after 0.9s idle (MaxConnectionIdle 300ms), state: IDLE
next call reconnects transparently, error: <nil> - state now: READY

Detecting the dead peer

After the forwarder was made dead, the server's handler returned from its wait about 2 seconds later: one second of idleness before the server sent a ping, one more second without an answer, then the connection was closed and the stream's context was cancelled. The handler could then clean up. Without keepalive, nothing would have happened until the operating system gave up on the connection, which can take from minutes to hours depending on settings.

The numbers in the lab are deliberately tiny. Production values are tens of seconds, for example Time: 30s, Timeout: 10s, which detects a dead peer in under a minute. Shorter intervals find failures faster and cost more pings and more false positives on slow links. Pick the detection time you can tolerate and set Time to about a third of it.

Recycling healthy connections

The second half of the program configures a server with MaxConnectionIdle: 300ms. After one call the client connection is READY. After 900 ms of silence the server has closed it, and the client reports IDLE: the channel is intact, the underlying connection is gone, and it will reconnect on the next call. The next call returned no error and the state went back to READY. This is how gRPC channels are meant to behave, and it is why you hold one ClientConn for the life of the process: the connection underneath comes and goes without your code noticing.

The settings

Client: keepalive.ClientParameters

grpc.WithKeepaliveParams(keepalive.ClientParameters{
	Time:                30 * time.Second, // ping after this much inactivity
	Timeout:             10 * time.Second, // wait this long for the reply
	PermitWithoutStream: false,            // ping even when no call is active?
})

Time has a minimum: grpc-go raises any value below 10 seconds to 10 seconds. PermitWithoutStream decides whether the client pings when it has no active calls. The default is false, which is usually right: an idle connection with no calls costs nothing and will be re-established when needed. Set it to true only when you want to keep a connection warm (to avoid handshake latency) or to detect a dead peer before the next call.

Server: keepalive.ServerParameters

grpc.KeepaliveParams(keepalive.ServerParameters{
	MaxConnectionIdle:     5 * time.Minute,  // close connections with no calls for this long
	MaxConnectionAge:      30 * time.Minute, // close any connection after this age (with jitter)
	MaxConnectionAgeGrace: 30 * time.Second, // let running calls finish after the age is reached
	Time:                  30 * time.Second, // ping idle clients
	Timeout:               10 * time.Second, // and close if they do not answer
})

Time and Timeout on the server detect dead clients, as in the lab. MaxConnectionIdle frees resources held by clients that have gone quiet. MaxConnectionAge is the quiet hero: it makes the server close every connection after a while, with a graceful GOAWAY so in-flight calls complete, which forces clients to reconnect. Reconnecting means re-resolving names and picking backends again, so new replicas get traffic and drained ones are left behind. Without it, a connection can live for weeks and pin a client to one backend through any number of deployments (load balancing). Add a few percent of jitter (grpc-go does) so a thousand clients do not all reconnect in the same second.

Server: keepalive.EnforcementPolicy

grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
	MinTime:             15 * time.Second, // pings more often than this are abuse
	PermitWithoutStream: false,            // pings on connections with no calls are abuse
})

This protects the server from clients that ping too often. The defaults are strict: MinTime is 5 minutes and PermitWithoutStream is false. If a client pings more often, the server counts a strike, and after a few strikes it closes the connection with a GOAWAY whose debug message is too_many_pings.

The too_many_pings trap

This is the classic keepalive failure. A team sets client Time: 30s and PermitWithoutStream: true to keep connections alive through a proxy, and does not touch the server. Now the client pings every 30 seconds, and the server's default policy says anything under 5 minutes is abuse. Within a couple of minutes connections start being closed. The client logs Server sent GOAWAY with debug data too_many_pings. Calls fail with Unavailable, retries reconnect, the cycle repeats and the symptom looks like random network trouble.

The rule: the server's EnforcementPolicy.MinTime must be no larger than the client's Time, and if clients ping without calls, the server must set PermitWithoutStream: true. Keep both sides' numbers next to each other in one document or one shared config, because they are only correct together. When a client receives too_many_pings, grpc-go doubles its keepalive interval for subsequent connections, which is why the problem can look intermittent.

Matching your infrastructure

Keepalive interacts with every box between the two processes. Load balancers and NAT gateways drop idle connections after a timeout (commonly in the range of one to several minutes; check your provider's documentation). If the idle timeout is 60 seconds, a ping every 30 seconds keeps the connection mapped, and a ping every 90 seconds does not. So choose Time below the smallest idle timeout on the path, and MaxConnectionIdle either above the proxies' timeout or you will see the proxy and the server cutting connections at different moments.

For long-lived streams, keepalive is the right tool, not a deadline on the stream. For short unary calls, you may not need client pings at all: calls have deadlines, and a dead connection fails the call quickly. The place keepalive earns its keep is streams and idle connections that you do not want to lose.

Observability and testing

Log connection state changes in a development build (GRPC_GO_LOG_SEVERITY_LEVEL=info and GRPC_GO_LOG_VERBOSITY_LEVEL=2) to see GOAWAY and state transitions. Count connections and their ages on the server. In tests, use a forwarder like the lab's to simulate silence, with small values for Time and Timeout, and assert that the handler's context is cancelled within the expected time. Test the client and server configuration together, with the real settings, so a mismatch produces too_many_pings in a test run and not in production. The settings are documented in the gRPC keepalive guide and in the keepalive package.