Timeouts Retries and Deadlines Across Go Service Calls

Most outages in distributed systems are not caused by a service going down. They are caused by a service getting slow, and by everything that depends on it responding with more work. Callers wait. Callers time out and retry. Retries add load to a service that was already struggling. Connections pile up, goroutines pile up, and a slowdown in one service becomes an outage in five.

The defences are old and simple: timeouts so waiting is bounded, deadlines so the bound is shared along the chain, retries done carefully, and a refusal to do work nobody is waiting for. This article shows how they fit together in Go with a three-service chain, including the measurement that most convinced me to write it: how many requests reach the failing service when each layer retries politely. It builds on context deadlines, retry with backoff and the circuit breaker. It ran on Go 1.25.6.

Timeouts, deadlines and budgets

These words get used loosely, so fix them.

A timeout is a duration: wait at most 2 seconds for this call. A deadline is a point in time: this must be finished by 12:00:03.250. The difference matters when work passes between services. If A gives B a 2 second timeout and B gives C its own 2 second timeout, the total can reach 4 seconds. If instead the client sets a deadline and every hop shrinks the remaining time as it passes it on, the whole chain fits in the original budget.

Go's context package already represents a deadline. context.WithTimeout computes the point in time from now plus a duration. What the standard library does not do for HTTP is send that deadline across the network. gRPC does: it puts the remaining time in a grpc-timeout header and the server derives its context from it. Over plain HTTP you choose a header, as the lab does with X-Deadline-Ms, and a middleware turns it back into a context. Send a remaining duration, not an absolute timestamp, because clocks on two machines disagree by more than you would like.

Which timeouts exist

An HTTP call has several phases, and each can hang separately, so http.Client.Timeout alone is not a full answer.

A sensible service client sets the first three tight (a connect timeout of a second or less within a data centre), uses a context deadline for the whole call, and keeps Client.Timeout as a backstop. The HTTP client timeout article shows the configuration. A common mistake is a long total timeout with no response-header timeout: the call hangs until the whole budget is spent even though the server never intended to answer.

On the server side, ReadHeaderTimeout, ReadTimeout, WriteTimeout and IdleTimeout protect you from slow clients (server timeouts). They are separate from the per-request deadline derived from the caller's budget.

What goes wrong with retries

A retry is a bet that a failure was temporary. When it is, retries turn errors into slightly slower successes. When the failure is overload, retries are more load. The danger is multiplication through layers.

The lab builds a chain: a client calls service A, A calls B, B calls C, and C always fails with 503 and counts the requests that reach it. A and B each retry up to three times, with a millisecond between attempts.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"strconv"
	"sync/atomic"
	"time"
)

const deadlineHeader = "X-Deadline-Ms" // remaining budget, in milliseconds, set by the caller

// withDeadline turns the caller's remaining budget into a context deadline for this handler.
func withDeadline(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if ms, err := strconv.Atoi(r.Header.Get(deadlineHeader)); err == nil {
			ctx, cancel := context.WithTimeout(r.Context(), time.Duration(ms)*time.Millisecond)
			defer cancel()
			r = r.WithContext(ctx)
		}
		next.ServeHTTP(w, r)
	})
}

// call performs one attempt and forwards the remaining budget so the next hop can respect it.
func call(ctx context.Context, url string) error {
	req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if dl, ok := ctx.Deadline(); ok {
		req.Header.Set(deadlineHeader, strconv.Itoa(int(time.Until(dl).Milliseconds())))
	}
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	io.Copy(io.Discard, resp.Body)
	if resp.StatusCode >= 500 {
		return fmt.Errorf("status %d", resp.StatusCode)
	}
	return nil
}

// retry makes up to attempts calls, but never starts one that cannot finish inside the caller's deadline.
func retry(ctx context.Context, attempts int, minUseful time.Duration, fn func(context.Context) error) error {
	var err error
	for i := 0; i < attempts; i++ {
		if dl, ok := ctx.Deadline(); ok && time.Until(dl) < minUseful {
			return fmt.Errorf("giving up before attempt %d, %v left: %w", i+1, time.Until(dl).Round(time.Millisecond), err)
		}
		if err = fn(ctx); err == nil {
			return nil
		}
		select {
		case <-time.After(time.Millisecond):
		case <-ctx.Done():
			return ctx.Err()
		}
	}
	return err
}

// chain builds A -> B -> C. C fails (or is slow) and counts every request that reaches it.
func chain(retriesA, retriesB int, minUseful, cDelay time.Duration) (a *httptest.Server, cHits *atomic.Int64, closeAll func()) {
	cHits = new(atomic.Int64)
	c := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		cHits.Add(1)
		select {
		case <-time.After(cDelay):
			http.Error(w, "boom", http.StatusServiceUnavailable)
		case <-r.Context().Done():
		}
	}))
	b := httptest.NewServer(withDeadline(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if err := retry(r.Context(), retriesB, minUseful, func(ctx context.Context) error { return call(ctx, c.URL) }); err != nil {
			http.Error(w, err.Error(), http.StatusBadGateway)
		}
	})))
	a = httptest.NewServer(withDeadline(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if err := retry(r.Context(), retriesA, minUseful, func(ctx context.Context) error { return call(ctx, b.URL) }); err != nil {
			http.Error(w, err.Error(), http.StatusBadGateway)
		}
	})))
	return a, cHits, func() { a.Close(); b.Close(); c.Close() }
}

func main() {
	// 1. Retries at every layer multiply. The client makes one request.
	for _, c := range []struct {
		label  string
		ra, rb int
	}{{"no retries anywhere", 1, 1}, {"3 attempts at B only", 1, 3}, {"3 attempts at A and B", 3, 3}} {
		a, hits, closeAll := chain(c.ra, c.rb, 0, 0)
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		err := call(ctx, a.URL)
		cancel()
		fmt.Printf("%-24s requests that reached C: %2d (client saw error: %v)\n", c.label, hits.Load(), err != nil)
		closeAll()
	}

	// 2. A slow dependency and a client budget of 300 ms. C needs 120 ms per attempt.
	a, hits, closeAll := chain(3, 3, 130*time.Millisecond, 120*time.Millisecond)
	start := time.Now()
	ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
	err := call(ctx, a.URL)
	cancel()
	took := time.Since(start)
	fmt.Printf("300ms budget, C needs 120ms: requests that reached C: %2d, client back within 400ms: %v, error: %v\n", hits.Load(), took < 400*time.Millisecond, err != nil)
	closeAll()
}

no retries anywhere      requests that reached C:  1 (client saw error: true)
3 attempts at B only     requests that reached C:  3 (client saw error: true)
3 attempts at A and B    requests that reached C:  9 (client saw error: true)
300ms budget, C needs 120ms: requests that reached C:  2, client back within 400ms: true, error: true

Reading the first three lines

With no retries, the one client request produced one request at C. With three attempts at B only, C saw three. With three attempts at both A and B, C saw nine. The client made one request, saw one error, and a service that was already failing received nine times the work.

Add a third layer with three attempts and it is 27. This is why "just add retries to the HTTP client" is a dangerous default when every service in the system uses the same client library: the retry policy is multiplied by the depth of the call graph, and the services most deeply nested take the largest hit exactly when they are least able to.

The remedies

Retry at one layer. Usually the edge: the caller closest to the user, which knows whether the whole operation is worth repeating. Internal hops fail fast and report. This is the simplest rule and eliminates the multiplication.

Use a retry budget. Allow retries only as a small fraction of recent traffic, say 10 percent of requests in the last minute. A token bucket that refills with each successful call and is drained by each retry gives that behaviour. When the dependency is failing, the bucket empties and retries stop, so the extra load is capped at 10 percent, not 200.

Do not retry what you should not. Retry connection failures before any request was sent, and 503 or 429 responses that carry Retry-After (and honour it). Do not retry 400 responses, which will fail again. Retry non-idempotent calls like POST /payments only with an idempotency key, otherwise you may do the work twice.

Back off with jitter. Wait longer after each failure, with a random component so a thousand clients do not retry in lockstep. A cap of a few seconds and "full jitter" (a random delay between zero and the exponential ceiling) is a standard choice, covered in retry with backoff.

Stop when a breaker is open. A circuit breaker refuses calls to a dependency that is clearly down, which turns a slow failure into a fast one and gives the dependency room to recover.

Deadlines stop wasted work

The second experiment in the lab changes the failure: C no longer fails quickly, it takes 120 ms per request and then returns 503. The client has a 300 ms budget. A and B both retry up to three times.

The rule in retry is: do not start an attempt if the remaining time is less than a threshold that reflects what an attempt needs. Here that is 130 ms. The last output line shows the result: only 2 requests reached C, the client had its error within 400 ms, and the error was an error, not a hang.

Walk through the timeline. B's first attempt starts at 0 ms with about 290 ms left, since A already used some time. It fails at 120 ms. B waits a millisecond and checks: roughly 170 ms left, above the 130 ms threshold, so it starts a second attempt. That fails at about 242 ms. Now 58 ms remain, below the threshold, so B returns an error immediately without a third attempt. A sees the error, checks its own remaining time, also below the threshold, and returns. The third attempt would have been cancelled at the 300 ms mark anyway, but it would have occupied C for 58 ms of work that nobody could use.

Two mechanisms cooperate here. The remaining budget travels in X-Deadline-Ms, and the withDeadline middleware turns it into the request context. And the retry helper consults ctx.Deadline() before each attempt. Without the first, B would not know the client's budget. Without the second, B would know and retry anyway.

When a request's context is cancelled, everything downstream should stop. HTTP servers cancel r.Context() when the client disconnects, and clients built with NewRequestWithContext abort when their context ends. In the lab, C's handler selects on r.Context().Done() so a cancelled request releases its goroutine and the connection straight away. This is what context cancellation is for, and the practical upshot is that every blocking operation in a handler (database query, HTTP call, channel wait) must take the context. A handler that ignores it keeps burning resources for a caller that left.

Setting the numbers

There is no formula, but a procedure works.

1. Start from the user. What is the longest a person or an upstream system will wait? That is the edge deadline, say 2 seconds.
2. Measure each dependency's latency distribution. Look at p99, not the median. If a dependency's p99 is 300 ms, a 100 ms timeout turns 1 percent of calls into errors.
3. Set each timeout above the dependency's p99 and below the caller's remaining budget. Calls closer to the leaves get shorter timeouts, because the deadline shrinks as it passes down.
4. Cap attempts so the sum fits. Three attempts of 300 ms plus backoff cannot fit in 500 ms; the threshold check will drop the later attempts.
5. Order the layers. Client timeout > gateway timeout > service timeout > database timeout. If the order is reversed, the outer layer gives up first and the inner layer keeps working on abandoned requests. The API gateway article makes the same point about the edge.

Write the numbers down in one place per service, review them with the latency graphs, and revisit them when a dependency changes.

Related defences

Load shedding. When a service is over capacity, answer 503 quickly for the excess rather than queueing everything. Fast rejections keep the work in progress healthy and make the retry budget do its job. See load shedding.

Hedged requests. For latency-sensitive reads with a spiky tail, send a second request to another replica after the p95 delay and take the first answer. It trades extra load for lower tail latency, so it needs the same budget discipline and only suits idempotent calls.

Bulkheads. Give each dependency its own concurrency limit, so a slow one cannot consume every goroutine and connection. A semaphore per dependency is enough (semaphore pattern).

Observability. Count attempts, not just requests, and tag metrics with the attempt number. A graph that shows retries as a share of traffic is the early warning for the multiplication problem, long before an outage. Log when a deadline was exceeded versus when the client cancelled, since they point to different faults.

A retry budget in Go

The budget idea is short enough to write in a few lines. Each successful call adds a fraction of a token; each retry spends a whole one; the bucket has a ceiling.

type RetryBudget struct {
	mu     sync.Mutex
	tokens float64
	max    float64
	ratio  float64 // tokens earned per success, for example 0.1
}

func (b *RetryBudget) Success() {
	b.mu.Lock()
	b.tokens = min(b.tokens+b.ratio, b.max)
	b.mu.Unlock()
}

func (b *RetryBudget) AllowRetry() bool {
	b.mu.Lock()
	defer b.mu.Unlock()
	if b.tokens < 1 {
		return false
	}
	b.tokens--
	return true
}

With a ratio of 0.1, ten successful calls earn one retry. Under normal conditions the occasional transient failure finds a token waiting and gets retried. When the dependency is down, there are no successes to refill the bucket, so after the initial reserve is spent the retries stop, and the extra load is bounded by the reserve instead of growing with the failure. Create one budget per dependency, not one per process, so a failing payments service does not use up the budget for search.

Check AllowRetry after the deadline check and the idempotency check. Order matters for the metrics: count a refused retry separately from a failed call, because a rising count of refused retries is the earliest sign that a dependency is in trouble.

What to record

When a call chain misbehaves, the questions you ask are always the same. How many attempts did this request take? What was the remaining budget at each hop? Which layer gave up first? Put the attempt number and the remaining budget in your structured logs and trace spans as fields, and propagate the request id through every hop. A log line such as attempt=2 budget_ms=171 dependency=c outcome=503 is worth more than a paragraph of prose in a postmortem. The logging side is covered in API logging without leaking secrets and the tracing side in tracing requests across APIs.

Testing it

Make the timing a parameter, as the lab does with cDelay and minUseful, and use real HTTP servers from httptest so the behaviour includes the transport. Assert on counts at the leaf (how many requests reached C), not only on what the client saw. That is how the amplification shows up. For timing assertions, leave generous margins, such as "returned in under 400 ms" for a 300 ms budget, and prefer counting events to measuring durations. The synctest package makes timer-heavy code deterministic; see the testing time-dependent code article.

The gRPC equivalents of everything here, deadlines propagating automatically and retry policy in the service config, are in propagating deadlines in gRPC and gRPC retries. The Google SRE book chapter on handling overload and the chapter on addressing cascading failures describe the retry budget and the failure pattern in more depth than there is room for here.