gRPC Deadlines and Timeouts That Propagate Across Services

Most outages in a service graph are a story about time. A slow database makes one service slow; its callers wait; their callers wait; threads and connections pile up; everything times out at once. The standard defence is a deadline: a point in time after which the answer is worthless, passed along with the request so every service can stop as soon as it is pointless to continue.

gRPC has deadlines built in, and they propagate across process boundaries without extra work, provided you let them. This article shows that with a three-service chain, then shows the small mistake that breaks it. It extends unary RPCs and the general treatment of timeouts and retries across services, and it matters for retries and graceful shutdown. It ran on Go 1.25.6.

How a deadline travels

On the client, context.WithTimeout(ctx, 600*time.Millisecond) creates a context with a deadline 600 ms from now. When you make a gRPC call with it, grpc-go sends the remaining time in an HTTP/2 header called grpc-timeout (for example 600m). It sends a duration, not an absolute time, because clocks on different machines disagree.

On the server, the library reads that header and creates the handler's context with a deadline set to that duration from the moment the request arrived. In the handler, ctx.Deadline() returns it. If the deadline passes, ctx.Done() closes. And if the client cancels or disconnects first, the context is cancelled.

Because the server context already carries the deadline, a handler that makes its own gRPC call with the same context passes the remaining budget on: the next call sends grpc-timeout with whatever is left. No code is required beyond using the right variable.

The experiment

Three services in a chain: A calls B calls C. A does 20 ms of work, B 60 ms, C 120 ms. Each records how much deadline it saw when its handler started. The program runs three scenarios.

syntax = "proto3";
package relay.v1;
option go_package = "example.test/lab/relaypb";

service Relay {
  rpc Call(CallRequest) returns (CallReply);
}

message CallRequest { string path = 1; }
message CallReply { string path = 1; }

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sync"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/status"

	pb "example.test/lab/relaypb"
)

// hop is one service in a chain. It spends `work` doing its own job, then calls the next hop (if any).
type hop struct {
	pb.UnimplementedRelayServer
	name       string
	work       time.Duration
	next       pb.RelayClient
	dropCtx    bool // the bug: call downstream with a fresh context, forgetting the caller's deadline
	log        *sync.Map
	workedPast *sync.Map // set when a hop finished its work after its caller had already given up
}

func remaining(ctx context.Context) string {
	dl, ok := ctx.Deadline()
	if !ok {
		return "none"
	}
	return time.Until(dl).Round(10 * time.Millisecond).String()
}

func (h *hop) Call(ctx context.Context, req *pb.CallRequest) (*pb.CallReply, error) {
	h.log.Store(h.name, remaining(ctx))
	select {
	case <-time.After(h.work):
	case <-ctx.Done():
		return nil, status.FromContextError(ctx.Err()).Err()
	}
	path := req.GetPath() + h.name
	if h.next == nil {
		return &pb.CallReply{Path: path}, nil
	}
	callCtx := ctx
	if h.dropCtx {
		callCtx = context.Background() // loses the deadline and the cancellation
	}
	rep, err := h.next.Call(callCtx, &pb.CallRequest{Path: path + ">"})
	if err != nil {
		return nil, err
	}
	if ctx.Err() != nil {
		h.workedPast.Store(h.name, true)
	}
	return rep, nil
}

func start(h *hop) (pb.RelayClient, func()) {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	srv := grpc.NewServer()
	pb.RegisterRelayServer(srv, h)
	go srv.Serve(lis)
	conn, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	return pb.NewRelayClient(conn), func() { conn.Close(); srv.Stop() }
}

func run(label string, dropCtx bool, deadline time.Duration) {
	var seen, past sync.Map
	c := &hop{name: "C", work: 120 * time.Millisecond, log: &seen, workedPast: &past}
	cc, stopC := start(c)
	defer stopC()
	b := &hop{name: "B", work: 60 * time.Millisecond, next: cc, dropCtx: dropCtx, log: &seen, workedPast: &past}
	bc, stopB := start(b)
	defer stopB()
	a := &hop{name: "A", work: 20 * time.Millisecond, next: bc, log: &seen, workedPast: &past}
	ac, stopA := start(a)
	defer stopA()

	ctx, cancel := context.WithTimeout(context.Background(), deadline)
	defer cancel()
	rep, err := ac.Call(ctx, &pb.CallRequest{})
	fmt.Printf("%s (client deadline %v)\n", label, deadline)
	if err != nil {
		fmt.Printf("  client got: %s\n", status.Code(err))
	} else {
		fmt.Printf("  client got: OK, path %s\n", rep.GetPath())
	}
	for _, n := range []string{"A", "B", "C"} {
		v, _ := seen.Load(n)
		fmt.Printf("  %s saw a deadline of %v when its handler started\n", n, v)
	}
	time.Sleep(400 * time.Millisecond) // give a hop that ignored the deadline time to finish its useless work
	for _, n := range []string{"A", "B", "C"} {
		if _, ok := past.Load(n); ok {
			fmt.Printf("  %s kept working after its caller had given up\n", n)
		}
	}
}

func main() {
	run("1. a budget big enough for the whole chain", false, 600*time.Millisecond)
	run("2. a budget too small: C needs more than is left", false, 150*time.Millisecond)
	run("3. the same budget, but B calls C with a fresh context", true, 150*time.Millisecond)
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

1. a budget big enough for the whole chain (client deadline 600ms)
  client got: OK, path A>B>C
  A saw a deadline of 600ms when its handler started
  B saw a deadline of 570ms when its handler started
  C saw a deadline of 510ms when its handler started
2. a budget too small: C needs more than is left (client deadline 150ms)
  client got: DeadlineExceeded
  A saw a deadline of 150ms when its handler started
  B saw a deadline of 130ms when its handler started
  C saw a deadline of 70ms when its handler started
3. the same budget, but B calls C with a fresh context (client deadline 150ms)
  client got: DeadlineExceeded
  A saw a deadline of 150ms when its handler started
  B saw a deadline of 130ms when its handler started
  C saw a deadline of none when its handler started
  B kept working after its caller had given up

Scenario 1: enough budget

The client allows 600 ms. A saw 600 ms when it started. It spent 20 ms, then called B, which saw 580 ms. B spent 60 ms and called C, which saw 510 ms (the numbers are rounded to 10 ms; the extra few milliseconds are network and scheduling time). The whole chain needs about 200 ms, so it succeeds and the client gets the path A>B>C.

The point is the arithmetic: the budget shrinks as it moves. Nobody configured a timeout for B or C. They inherited what was left.

Scenario 2: not enough budget

The client allows 150 ms. A saw 150, B saw 130, C saw 60. C needs 120 ms and has 60. The select in the lab handler waits for either the work or ctx.Done(), so C stops when the deadline arrives, returns a DeadlineExceeded status, B returns the same error upward, and the client receives DeadlineExceeded. Time spent after the deadline is zero: every service gave up at about the same moment, instead of each running its own full timeout.

Compare what would happen with fixed per-hop timeouts. If each service gave its downstream call 500 ms regardless, a client that had given up after 150 ms would still leave B waiting 500 ms for C, holding goroutines, connections and memory for a result nobody can use. Under load, that waste is the difference between slow and down.

Scenario 3: the lost deadline

Here B calls C with context.Background(). This is a common bug and a natural one to write: a developer wants the downstream call to complete even if the caller leaves (to finish a write, say), or just passes the wrong variable. The effect is in the output: C saw a deadline of none. It ran for its full 120 ms. The client had long since received DeadlineExceeded, but B stayed blocked waiting for C and the printed note shows that B kept going after its caller had given up.

In this tiny chain the cost is 100 ms of useless work. In a real system with a slow downstream and many requests, it is a pile of in-flight work that cannot be cancelled, which is how a service under stress gets worse, not better.

Rules that follow

Always pass the incoming context to outgoing calls. In every handler, the ctx argument is the context of the call. Pass it to database queries, to HTTP clients and to other gRPC calls. Reaching for context.Background() or context.TODO() inside a request path is almost always the bug.

Deliberately detach only when you mean it. Some work must outlive the request: writing an audit record, publishing an event, committing a compensating action. For that, use context.WithoutCancel(ctx) (Go 1.21), which keeps the context's values (trace ids) but drops cancellation and deadline, and give the detached work its own timeout. Write a comment saying why. A bare Background() loses the values as well.

Set a deadline at the edge. The first service that receives a request from a user or from outside the system decides the overall budget. A client that never sets a deadline hands every downstream service an unbounded one. A server-side interceptor can impose a default when none arrives (interceptors).

Leave headroom. When a service calls downstream with several attempts, it should not give the whole remaining budget to the first attempt. Divide it, or reserve time for its own work after the call returns. A service that uses 100 percent of its budget waiting on a dependency has none left to build the response.

Handle DeadlineExceeded precisely. It means the operation may or may not have completed on the server. For reads that is harmless. For a write, you cannot retry blindly; use idempotency keys (idempotency). Do not translate it to Internal; keep the code so callers and retry policies can tell time-outs from bugs.

Check the deadline before expensive work. At the start of a handler, a quick if time.Until(deadline) < minimumUseful { return DeadlineExceeded } avoids starting something that cannot finish. The retry helper in timeouts and retries across services used the same trick.

Other edges: databases and HTTP

The same context should flow into everything else a handler touches. database/sql's QueryContext and pgx's context-aware methods cancel the query when the context ends (cancelling database queries). HTTP clients built with NewRequestWithContext abort. A deadline that stops at the edge of the gRPC world and does not reach the database leaves slow queries running after the answer is gone.

Client-side defaults

On the client, a missing deadline is the commonest cause of hung programs. Two defences. A unary client interceptor that sets a default deadline when the context has none (shown in the interceptors article), and a code review habit of asking, for every outgoing call, "where does the deadline come from?". For long-lived streams, a total deadline is the wrong tool; use keepalive and application heartbeats instead (keepalive).

Testing deadline propagation

The lab is the test. Build a chain of in-process services with known delays, call it with a short deadline and assert three things: the client sees DeadlineExceeded, each hop saw a smaller remaining deadline than the previous one, and no hop kept working past the deadline (a flag set when the handler completes its work after ctx.Err() != nil). Run it in CI against your real handler wiring, not a copy, so the day someone introduces context.Background() in a handler, the test fails. The behaviour is specified in the gRPC deadlines guide and the HTTP/2 mapping in the gRPC over HTTP/2 specification.