gRPC Graceful Shutdown in Go Servers Without Dropped Calls

Every deployment restarts servers. If a restart drops the calls that are in flight, users see errors with each release, and teams respond by deploying less often, which is the wrong lesson. gRPC has the pieces for a clean exit; the work is putting them in the right order with the right timeouts. This article does that in Go, shows the difference between Stop and GracefulStop with a call caught in the middle, and covers the two cases that make graceful shutdown hang or fail: streams that never end, and load balancers that keep sending traffic to a pod that is leaving.

It builds on health checking and keepalive, and is the gRPC counterpart of graceful shutdown for HTTP servers. It ran on Go 1.25.6.

Stop and GracefulStop

Server.Stop() closes all listeners and connections immediately. Calls in flight are cancelled; their clients see an Unavailable error. Server.GracefulStop() does the polite version: it closes the listeners, tells every connection to stop accepting new calls (an HTTP/2 GOAWAY), waits for the running calls to finish, and then closes the connections. It blocks until all that is done.

The lab starts a slow Run call that takes 400 ms and stops the server 100 ms in.

syntax = "proto3";
package job.v1;
option go_package = "example.test/lab/jobpb";

service Jobs {
  rpc Run(RunRequest) returns (RunReply);
}

message RunRequest { int32 millis = 1; string name = 2; }
message RunReply { string result = 1; }

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sync/atomic"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/status"

	pb "example.test/lab/jobpb"
)

type jobs struct {
	pb.UnimplementedJobsServer
	finished atomic.Int64
}

func (j *jobs) Run(ctx context.Context, r *pb.RunRequest) (*pb.RunReply, error) {
	select {
	case <-time.After(time.Duration(r.GetMillis()) * time.Millisecond):
		j.finished.Add(1)
		return &pb.RunReply{Result: r.GetName() + " finished"}, nil
	case <-ctx.Done():
		return nil, status.FromContextError(ctx.Err()).Err()
	}
}

type env struct {
	srv  *grpc.Server
	impl *jobs
	hs   *health.Server
	c    pb.JobsClient
	hc   healthpb.HealthClient
}

func newEnv() *env {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	e := &env{srv: grpc.NewServer(), impl: &jobs{}, hs: health.NewServer()}
	pb.RegisterJobsServer(e.srv, e.impl)
	healthpb.RegisterHealthServer(e.srv, e.hs)
	go e.srv.Serve(lis)
	conn, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	e.c, e.hc = pb.NewJobsClient(conn), healthpb.NewHealthClient(conn)
	return e
}

// start launches a call in the background and waits until the server has it in flight.
func (e *env) start(name string, millis int32) <-chan string {
	out := make(chan string, 1)
	go func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		r, err := e.c.Run(ctx, &pb.RunRequest{Name: name, Millis: millis})
		if err != nil {
			out <- name + ": " + status.Code(err).String()
			return
		}
		out <- name + ": OK, " + r.GetResult()
	}()
	time.Sleep(100 * time.Millisecond)
	return out
}

func main() {
	// 1. Stop: closes connections at once. The call in flight is lost.
	e := newEnv()
	inflight := e.start("job-1", 400)
	e.srv.Stop()
	fmt.Println("Stop():                      ", <-inflight, "| jobs finished on the server:", e.impl.finished.Load())

	// 2. GracefulStop: refuses new calls, lets running ones finish.
	e = newEnv()
	inflight = e.start("job-2", 400)
	done := make(chan struct{})
	go func() { e.srv.GracefulStop(); close(done) }()
	time.Sleep(50 * time.Millisecond)
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	_, err := e.c.Run(ctx, &pb.RunRequest{Name: "late", Millis: 10})
	cancel()
	fmt.Println("GracefulStop(), new call:    ", "late:", status.Code(err))
	fmt.Println("GracefulStop(), running call:", <-inflight, "| jobs finished on the server:", e.impl.finished.Load())
	<-done

	// 3. The health service tells load balancers to move away before the server stops listening.
	e = newEnv()
	e.hs.SetServingStatus("", healthpb.HealthCheckResponse_SERVING)
	r, _ := e.hc.Check(context.Background(), &healthpb.HealthCheckRequest{})
	fmt.Println("before shutdown, health:     ", r.GetStatus())
	e.hs.Shutdown()
	r, _ = e.hc.Check(context.Background(), &healthpb.HealthCheckRequest{})
	fmt.Println("after hs.Shutdown(), health: ", r.GetStatus())
	e.srv.GracefulStop()

	// 4. A call that never finishes would make GracefulStop wait forever; a deadline on the wait falls back to Stop.
	e = newEnv()
	inflight = e.start("stuck", 60000)
	start := time.Now()
	stopped := make(chan struct{})
	go func() { e.srv.GracefulStop(); close(stopped) }()
	select {
	case <-stopped:
		fmt.Println("stuck call: GracefulStop returned by itself")
	case <-time.After(500 * time.Millisecond):
		e.srv.Stop() // out of patience: cancel what is left
		<-stopped
		fmt.Printf("stuck call: GracefulStop gave up after the 500ms budget (returned within 1s: %v), the call ended as: %s\n",
			time.Since(start) < time.Second, <-inflight)
	}
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

Stop():                       job-1: Unavailable | jobs finished on the server: 0
GracefulStop(), new call:     late: Unavailable
GracefulStop(), running call: job-2: OK, job-2 finished | jobs finished on the server: 1
before shutdown, health:      SERVING
after hs.Shutdown(), health:  NOT_SERVING
stuck call: GracefulStop gave up after the 500ms budget (returned within 1s: true), the call ended as: stuck: Unavailable

What the run shows

Stop loses the work. The call in flight ended with Unavailable, and the server's counter of finished jobs is 0. The handler's context was cancelled, and the work did not complete. For a read that is a retry; for a write it is a mess, since the client does not know whether the write happened.

GracefulStop completes it. The same call, with GracefulStop running 100 ms in, returned OK, job-2 finished, and the counter shows 1. The server kept running until the call was done, then stopped.

New calls are refused during the drain. While GracefulStop was waiting, a new call named late got Unavailable straight away. The server does not accept new work once it has begun to drain. Clients with a retry policy on Unavailable retry against another backend and never notice (retries).

The health signal comes first

GracefulStop refuses new calls at the moment you call it. Between that moment and the moment load balancers learn about it, requests are still being routed to this server and they fail. In a rolling deploy that gap shows up as a burst of errors on every restart.

The fix is to tell the world before you stop. The health service (health checking) has a status the balancer can watch. The third block of the lab shows the call: hs.Shutdown() sets every service to NOT_SERVING, and a check afterwards confirms it. When load balancers, Kubernetes readiness probes or client-side health checks see NOT_SERVING, they stop sending new traffic. Then, after a short wait for that to take effect, you call GracefulStop, and by that point few if any new calls arrive.

How long to wait? Long enough for the signal to propagate: for Kubernetes, the readiness probe period plus the time for endpoints to update, typically 5 to 15 seconds. It is a real cost, and it is what removes the error burst.

The full sequence

On SIGTERM, which is what Kubernetes and most process managers send:

1. Mark NOT_SERVING (hs.Shutdown()).
2. Wait a configured drain delay for load balancers to stop routing.
3. GracefulStop() in a goroutine.
4. Wait for it with a deadline. If it finishes, you are done. If the deadline passes, call Stop().
5. Exit.

In code:

func run(srv *grpc.Server, hs *health.Server, drain, budget time.Duration) {
	sig := make(chan os.Signal, 1)
	signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT)
	<-sig

	hs.Shutdown()
	time.Sleep(drain)

	done := make(chan struct{})
	go func() { srv.GracefulStop(); close(done) }()
	select {
	case <-done:
	case <-time.After(budget):
		srv.Stop()
		<-done
	}
}

signal.NotifyContext is a neat alternative to the manual channel. Keep drain + budget below the platform's kill timeout (terminationGracePeriodSeconds in Kubernetes, 30 seconds by default), or the platform will send SIGKILL in the middle.

The call that never ends

The fourth block of the lab starts a call that would run for 60 seconds. GracefulStop waits for all calls to finish, so by itself it would wait a minute. The lab waits 500 ms for it and then calls Stop(); the stuck call ends as Unavailable, and the whole shutdown completed in well under a second. That is the reason for the budget in step 4: GracefulStop has no timeout of its own, so a hung call or a stream that never ends keeps the process alive until something kills it.

Pick the budget from the longest call you consider legitimate. Most unary calls finish in seconds; give them 10 to 25. If you have calls that run longer, they need a design for being interrupted: a way to resume on another instance.

Streams are the hard case

A server-streaming call or a bidirectional stream can last hours, and GracefulStop waits for each of them. A subscription stream will hold your shutdown until the budget expires and then be cut off with an error. Better to end streams cleanly yourself, before calling GracefulStop:

• Keep a registry of active streams, or a shared "shutting down" channel that every long-running handler selects on.
• When shutdown starts, close that channel. Each stream handler returns a clean status that tells the client to reconnect elsewhere (Unavailable, or a documented "server is draining" code), after sending whatever final message your protocol defines.
• Clients treat that status as a routine reconnect, with backoff and jitter, and resume from their last sequence number (streaming).

This turns a hard cut at the budget into an orderly migration, and it spreads the reconnect storm over the time you allow.

Closing everything else

The gRPC server is only one thing that has to stop. The order matters: stop accepting new calls first, drain, and only then close the resources the handlers use, such as the database pool, caches and message producers. Closing the database before the handlers finish turns graceful into a series of sql: database is closed errors. Flush logs and traces last (tracing), so the final minutes are visible.

If the same process also serves HTTP (metrics, a REST gateway), shut the servers down together with a shared context and the same deadline; http.Server.Shutdown plays the same role as GracefulStop.

Testing shutdown

The lab is already the test. Start a server, put a slow call in flight, begin the shutdown and assert: the call completes with OK, a call started after the drain begins is refused, the health status flipped first, and a never-ending call is cut after the budget. Run it in CI, and also on a staging cluster: deploy a new version while a load generator runs and check the error rate during the rollout. A clean exit that has never been tested under load is a hope. The behaviours of both stop methods are documented in the grpc package, and the pod lifecycle timings in the Kubernetes documentation.