Retries and Backoff for the Go HTTP Client Done Safely
Add retries to a Go HTTP client as a RoundTripper with exponential backoff and jitter, Retry-After, body replay and a rule for which requests are safe.
A retry is a small piece of code with a large blast radius. Done well, it hides the occasional 503 and connection reset that any network produces, and a user never sees them. Done badly, it doubles a payment, turns a brief outage into a long one by hammering a struggling server, or keeps a cancelled request alive for a minute. The mechanics are simple. The hard part is deciding what is safe to repeat, and how long to wait.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
This article builds a retrying RoundTripper, puts it through the cases that matter, and keeps the sleeping testable by injecting it. It follows from the client defaults and the general retry theory in retry with backoff, and it is part of the web cluster that ends in a production HTTP server. Everything ran on Go 1.25.6 with the standard library only.
Why a RoundTripper
You can retry at three layers: around every call site, in a helper function, or inside the client. The first duplicates the logic. The second works but only for code that uses the helper. A RoundTripper wrapper sits inside http.Client, so every request made through that client gets the policy, including requests made by libraries you did not write, such as an SDK that accepts an *http.Client. It sees each attempt's raw result, can inspect the status, and can replay the request. The rule it must follow is documented on the interface: a RoundTripper must not modify the request, so retries clone it.
What is safe to repeat
The question is not "did it fail" but "might it have already worked". A connection refused before any byte was written is safe to retry for any method, since the server never saw the request. A timeout after the request was sent is not: the server may have processed it and been slow to answer. For that reason the policy is conservative:
- Methods.
GET,HEAD,OPTIONS,PUTandDELETEare idempotent by definition, so repeating them gives the same server state.POSTandPATCHare not, unless the request carries anIdempotency-Keyheader that the server uses to deduplicate, as built in idempotency keys in APIs. - Statuses. 429 (too many requests), 502, 503 and 504 mean the request was probably not processed or can safely be asked again. Other 5xx codes, 500 especially, often indicate a bug that a retry will repeat; some teams retry them once. 4xx errors, other than 429 and sometimes 408, are the client's fault and never improve with repetition.
- Errors. Network errors are retried unless the context is done. Errors from the TLS layer about certificates are not worth retrying.
Backoff and jitter
Waiting a fixed second between attempts is the beginning, not the end. If a server fails for a minute and a thousand clients all retry on the same schedule, they arrive together each time, which is called a thundering herd. Two additions fix it. Exponential growth (100 ms, 200 ms, 400 ms and so on, up to a cap) gives a struggling server increasing breathing room. Jitter (a random amount added or chosen within the delay) spreads the clients apart. The full jitter variant picks a random value between zero and the exponential delay, and it has the lowest total load in the analysis AWS published. I use "equal jitter": half the delay is fixed and half random, so there is always some real wait.
The implementation
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync/atomic"
"time"
)
type Retry struct {
Next http.RoundTripper
Attempts int // total tries, including the first
Base time.Duration // delay before the second try
Cap time.Duration // longest single wait
Sleep func(ctx context.Context, d time.Duration) error
Jitter func(d time.Duration) time.Duration // returns a value in [0, d]
}
func sleep(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-t.C:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func idempotent(r *http.Request) bool {
switch r.Method {
case http.MethodGet, http.MethodHead, http.MethodOptions, http.MethodPut, http.MethodDelete:
return true
}
return r.Header.Get("Idempotency-Key") != ""
}
func retryableStatus(code int) bool {
return code == http.StatusTooManyRequests || code == http.StatusBadGateway ||
code == http.StatusServiceUnavailable || code == http.StatusGatewayTimeout
}
// retryAfter reads the header as seconds or as an HTTP date.
func retryAfter(res *http.Response) (time.Duration, bool) {
v := res.Header.Get("Retry-After")
if v == "" {
return 0, false
}
if secs, err := strconv.Atoi(v); err == nil && secs >= 0 {
return time.Duration(secs) * time.Second, true
}
if t, err := http.ParseTime(v); err == nil {
return max(0, time.Until(t)), true
}
return 0, false
}
func (t *Retry) RoundTrip(req *http.Request) (*http.Response, error) {
canReplay := req.Body == nil || req.Body == http.NoBody || req.GetBody != nil
for attempt := 1; ; attempt++ {
r := req
if attempt > 1 && req.GetBody != nil {
body, err := req.GetBody()
if err != nil {
return nil, err
}
r = req.Clone(req.Context())
r.Body = body
}
res, err := t.Next.RoundTrip(r)
again := idempotent(req) && canReplay && attempt < t.Attempts && req.Context().Err() == nil
var wait time.Duration
switch {
case !again:
return res, err
case err != nil:
wait = t.backoff(attempt)
case retryableStatus(res.StatusCode):
wait = t.backoff(attempt)
if d, ok := retryAfter(res); ok {
wait = min(d, t.Cap) // the server's number wins, within our cap
}
io.Copy(io.Discard, io.LimitReader(res.Body, 4096))
res.Body.Close()
default:
return res, err
}
if err := t.Sleep(req.Context(), wait); err != nil {
return nil, err
}
}
}
func (t *Retry) backoff(attempt int) time.Duration {
d := min(t.Base<<(attempt-1), t.Cap)
return d/2 + t.Jitter(d/2)
}
func main() {
var hits atomic.Int64
var failFirst atomic.Int64
var lastBody atomic.Value
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits.Add(1)
body, _ := io.ReadAll(r.Body)
lastBody.Store(string(body))
switch r.URL.Path {
case "/flaky":
if failFirst.Add(-1) >= 0 {
http.Error(w, "busy", http.StatusServiceUnavailable)
return
}
case "/limited":
w.Header().Set("Retry-After", "2")
if failFirst.Add(-1) >= 0 {
http.Error(w, "slow down", http.StatusTooManyRequests)
return
}
case "/missing":
http.NotFound(w, r)
return
}
fmt.Fprint(w, "done")
}))
defer srv.Close()
var waits []time.Duration
rt := &Retry{
Next: http.DefaultTransport, Attempts: 4, Base: 100 * time.Millisecond, Cap: 5 * time.Second,
Sleep: func(ctx context.Context, d time.Duration) error { waits = append(waits, d); return ctx.Err() },
Jitter: func(d time.Duration) time.Duration { return d }, // the top of the range, so the output is stable
}
client := &http.Client{Transport: rt}
do := func(label, method, path, body string, hdr ...string) {
hits.Store(0)
waits = nil
req, _ := http.NewRequest(method, srv.URL+path, strings.NewReader(body))
for i := 0; i+1 < len(hdr); i += 2 {
req.Header.Set(hdr[i], hdr[i+1])
}
res, err := client.Do(req)
status := "error: " + fmt.Sprint(err)
if err == nil {
res.Body.Close()
status = strconv.Itoa(res.StatusCode)
}
fmt.Printf("%-34s status=%-4s attempts=%d waits=%v\n", label, status, hits.Load(), waits)
}
failFirst.Store(2)
do("GET, two 503s then success", "GET", "/flaky", "")
failFirst.Store(1)
do("GET, 429 with Retry-After: 2", "GET", "/limited", "")
failFirst.Store(1)
do("POST without a key", "POST", "/flaky", "order=1")
failFirst.Store(1)
do("POST with Idempotency-Key", "POST", "/flaky", "order=1", "Idempotency-Key", "k-1")
fmt.Printf(" body the server saw on the last attempt: %q\n", lastBody.Load())
do("GET a missing page", "GET", "/missing", "")
failFirst.Store(100)
do("GET, server stays down", "GET", "/flaky", "")
// Cancelling stops the retries at the first wait.
ctx, cancel := context.WithCancel(context.Background())
cancel()
failFirst.Store(100)
hits.Store(0)
req, _ := http.NewRequestWithContext(ctx, "GET", srv.URL+"/flaky", nil)
_, err := client.Do(req)
fmt.Printf("%-34s attempts=%d err=%v\n", "context already cancelled", hits.Load(), errors.Is(err, context.Canceled))
// A transport-level failure: nothing is listening.
srv.Close()
hits.Store(0)
waits = nil
_, err = client.Get(srv.URL)
fmt.Printf("%-34s waits=%v err present=%v\n", "GET, connection refused", waits, err != nil)
}GET, two 503s then success status=200 attempts=3 waits=[100ms 200ms]
GET, 429 with Retry-After: 2 status=200 attempts=2 waits=[2s]
POST without a key status=503 attempts=1 waits=[]
POST with Idempotency-Key status=200 attempts=2 waits=[100ms]
body the server saw on the last attempt: "order=1"
GET a missing page status=404 attempts=1 waits=[]
GET, server stays down status=503 attempts=4 waits=[100ms 200ms 400ms]
context already cancelled attempts=0 err=true
GET, connection refused waits=[100ms 200ms 400ms] err present=trueRead the output row by row, because each row is one rule. Two 503 responses followed by success took three attempts, with waits of 100 and 200 ms (the jitter function returned the top of its range so the numbers are stable; real jitter makes them vary between roughly half and the full value). The 429 carried Retry-After: 2, and the wait was two seconds, not the 100 ms backoff would have picked: the server's instruction replaces our guess, capped by Cap so a misbehaving server cannot park us for an hour.
A POST without a key was not retried: one attempt, the 503 returned to the caller. The same POST with an Idempotency-Key was retried, and the server saw the full body on the second attempt, which is the GetBody replay at work. http.NewRequest sets GetBody automatically for *bytes.Reader, *bytes.Buffer and *strings.Reader bodies; for a body from a file or a pipe you must provide it, or canReplay is false and the request is never retried. The 404 was returned immediately. When the server stayed down, the loop stopped at four attempts rather than spinning, with the waits growing and clamped by the cap. A context that was already cancelled made no attempt at all, because the transport refuses to send a request whose context is done, and the loop's own check stops any retry after a cancellation that arrives mid-flight.
Details that bite later
Total time. Four attempts with a 10 second timeout each can take 40 seconds plus the waits. Decide the budget for the whole operation first, put it on the context, and let the loop stop when it expires. Per-attempt timeouts (ResponseHeaderTimeout on the transport) then keep each try short while the context bounds the total, which is the combination described in the transport settings article.
Retry amplification. If three layers of services each retry three times, one user request can become 27 requests at the bottom when the database is struggling. Retry at one layer, usually the outermost one that owns the user's deadline, and let inner layers fail fast. Some systems add a retry budget, allowing retries only up to ten percent of recent traffic, which stops a retry storm from forming at all.
Breakers. Retries and circuit breakers are complements: retries mask a short fault, and a breaker stops sending when the fault is not short. Put the breaker outside the retry loop, so the whole retried call counts as one failure.
Observability. Log each retry at debug level with the attempt number, reason and wait, and export a counter. A service that is retrying constantly is unhealthy even if every call eventually succeeds, and without the counter nobody notices until the latency graph changes.
The specification for Retry-After is in RFC 9110 section 10.2.3, and the AWS Architecture Blog on exponential backoff and jitter has the simulations behind the jitter advice.
