Test Graceful Shutdown in Go Services Reliably and Fast

Graceful shutdown is code that runs once, at the worst possible time, and nothing exercises it until a deploy goes wrong. The promise is simple: when the process is told to stop, it stops accepting new work, lets the work in progress finish, and exits within a bounded time. Each clause is a behaviour a test can check, and none of the checks needs a sleep.

This article tests a small Run function on Go 1.25.6 around http.Server. The same shape works for gRPC servers, queue consumers and background workers: a test that controls when the work finishes, then triggers the stop.

The function

package main

import (
	"context"
	"errors"
	"net"
	"net/http"
	"time"
)

// Run serves h on l until ctx ends, then shuts down within grace.
func Run(ctx context.Context, l net.Listener, h http.Handler, grace time.Duration) error {
	srv := &http.Server{Handler: h}
	errc := make(chan error, 1)
	go func() { errc <- srv.Serve(l) }()

	select {
	case err := <-errc:
		return err
	case <-ctx.Done():
	}
	sctx, cancel := context.WithTimeout(context.Background(), grace)
	defer cancel()
	if err := srv.Shutdown(sctx); err != nil {
		_ = srv.Close() // grace expired: cut the remaining connections
		return err
	}
	if err := <-errc; !errors.Is(err, http.ErrServerClosed) {
		return err
	}
	return nil
}

func main() {}

Run serves a handler on an existing listener until the context ends. Passing the listener in, rather than an address, is what makes it testable: the test opens 127.0.0.1:0 and learns the port. When the context is cancelled, Run calls srv.Shutdown with a deadline of grace. Shutdown closes the listener, so new connections are refused, then waits for active requests to finish. If the grace runs out, Run calls srv.Close to cut what remains and returns the error. A clean stop returns nil, after confirming that Serve returned http.ErrServerClosed, the sentinel for an orderly exit.

Two behaviours, two tests

package main

import (
	"context"
	"errors"
	"io"
	"net"
	"net/http"
	"testing"
	"time"
)

func start(t *testing.T, h http.Handler, grace time.Duration) (addr string, stop func() error) {
	t.Helper()
	l, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		t.Fatal(err)
	}
	ctx, cancel := context.WithCancel(t.Context())
	done := make(chan error, 1)
	go func() { done <- Run(ctx, l, h, grace) }()
	return l.Addr().String(), func() error { cancel(); return <-done }
}

func TestInFlightRequestFinishes(t *testing.T) {
	entered := make(chan struct{})
	release := make(chan struct{})
	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		close(entered)
		<-release
		io.WriteString(w, "finished")
	})
	addr, stop := start(t, h, 2*time.Second)

	body := make(chan string, 1)
	go func() {
		resp, err := http.Get("http://" + addr)
		if err != nil {
			body <- "error: " + err.Error()
			return
		}
		defer resp.Body.Close()
		b, _ := io.ReadAll(resp.Body)
		body <- string(b)
	}()
	<-entered // the request is now in flight

	stopped := make(chan error, 1)
	go func() { stopped <- stop() }()
	// Shutdown closes the listener first; poll until a new connection is refused.
	var err error
	for deadline := time.Now().Add(2 * time.Second); time.Now().Before(deadline); time.Sleep(5 * time.Millisecond) {
		var c net.Conn
		if c, err = net.DialTimeout("tcp", addr, 200*time.Millisecond); err != nil {
			break
		}
		c.Close()
	}
	t.Logf("new connection during shutdown refused: %v", err != nil)

	close(release)
	t.Logf("in-flight response: %q", <-body)
	t.Logf("Run returned: %v", <-stopped)
}

func TestGraceExpires(t *testing.T) {
	entered := make(chan struct{})
	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		close(entered)
		<-r.Context().Done() // a handler that only ends when its connection is cut
	})
	addr, stop := start(t, h, 100*time.Millisecond)
	go http.Get("http://" + addr)
	<-entered
	err := stop()
	t.Logf("Run returned: %v (deadline exceeded: %v)", err, errors.Is(err, context.DeadlineExceeded))
}

go test -count=1 -v . 2>&1 | grep -vE "^(=== RUN|=== PAUSE|=== CONT|PASS|ok)" | sed -E "s/ \([0-9.]+s\)//; s#^ +[a-z_]+\.go:[0-9]+: #    #"

    new connection during shutdown refused: true
    in-flight response: "finished"
    Run returned: <nil>
--- PASS: TestInFlightRequestFinishes
    Run returned: context deadline exceeded (deadline exceeded: true)
--- PASS: TestGraceExpires

start listens, runs Run in a goroutine, and returns the address and a stop function that cancels the context and returns what Run returned. The parent context is t.Context(), so a test that fails early still stops its server during cleanup.

The in-flight request finishes

The handler in TestInFlightRequestFinishes does two things the test controls. It closes entered when it starts, and it blocks on release until the test lets it go. A client goroutine sends a request, and the test waits for entered, which is the proof that the request is in flight. There is no time.Sleep to hope the request has arrived.

Then the test stops the server, in another goroutine because stop blocks until Run returns. It polls until a new connection is refused, with a two second ceiling. Shutdown closes the listener first, and the poll waits for that to be true, so the assertion refused: true is not racing the shutdown. Only then does the test close release. The client receives finished, and Run returns <nil>.

The order of events is the content of the test: the listener closed before the request ended, and the request still completed. A server that closed connections immediately would have failed the body check, and a server that kept accepting would have failed the refusal check.

The grace period holds

TestGraceExpires uses a handler that only returns when its request context ends, which happens when the connection is cut. The server has a 100 ms grace. Shutdown waits, the deadline passes, and it returns context deadline exceeded. Run then calls Close, which cuts the connection, so the handler ends and the test can finish. The recorded line reads Run returned: context deadline exceeded (deadline exceeded: true).

This test is the one that prevents the worst deploy behaviour: a stuck request holding the process open until the orchestrator sends SIGKILL. It proves that the bound exists and that the code reports exceeding it. Keep the grace short in the test so it costs a tenth of a second.

What this does not cover

Run is called with a context here. In main, that context comes from signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM), and the signal wiring is not tested by these tests. It is a few lines and is best verified once, by hand or with an end-to-end test that starts the binary, sends SIGTERM, and checks the exit code; that is the subject of the CLI testing article. Do not test the signal in-process by sending your own test binary a signal, because the testing package handles SIGINT itself.

Also outside the test: connections that were hijacked, such as WebSockets, which Shutdown does not wait for and RegisterOnShutdown exists to notify; keep-alive connections that are idle, which Shutdown closes; and background goroutines the handlers started, which the server does not track. A shutdown that matters needs a shutdown for each of those, and a test for each.

Variations for other servers

• gRPC. GracefulStop has the same role as Shutdown and blocks until RPCs finish. A stream that never ends blocks it forever, so the grace-expiry test calls Stop after a timer, as Run does. Use bufconn, from the gRPC test article, to avoid a real port.
• Queue consumers. Closing the input channel is the stop signal, and the test asserts that messages in progress are acknowledged, as described in testing consumers.
• Workers with a WaitGroup. Cancel the context, then wait for the group with a guard timeout, per the goroutine testing article.

Keeping the test fast and honest

The two tests take a fraction of a second between them, and the speed comes from three decisions. The grace period is a parameter, so the expiry test uses 100 ms instead of the 30 seconds production uses. The handler is controlled with channels, so no test waits for a duration it merely hopes is long enough. And the listener is on port 0, so the test never waits for a port to free up between runs.

Watch for the opposite temptations. A test that sleeps 100 ms "to let the shutdown start" will pass on your machine and fail on a loaded runner. A test that asserts the shutdown took less than some number of milliseconds measures the machine. Assert the order of events and the final state, and leave timing to the guard timeouts. If a test hangs, a -timeout of 30 seconds on the package turns it into a goroutine dump that names the blocked Shutdown call.

It is worth running the pair under -race -count=50 once after writing them. Shutdown code is where goroutines from Serve, Shutdown and the handlers meet, and a missing synchronisation shows up quickly under repetition, as the flaky tests guide describes.

Readiness and health

An orchestrator takes a pod out of rotation by failing its readiness probe, and a well-behaved service starts failing readiness at the moment shutdown begins, then waits a few seconds for the load balancer to notice before it stops accepting. That is two phases, and each is testable: a handler that reports not ready after the context is cancelled, and a delay between the flip and the Shutdown call. The delay is a configuration value, so the test sets it to zero.

The reference for the behaviour is the http.Server.Shutdown documentation, and the standard pattern is described in the signal.NotifyContext documentation. Related reading on the timeout side is context cancellation tests.