Retryable and Permanent Errors in Go and How to Tell Them

Retrying is the cheapest reliability feature there is, and the easiest to do badly. A retry loop that works on every error will hammer a service that is returning 403 because your token expired, will re-send a payment that already went through, and will keep a cancelled request alive long after the user left. The loop is a dozen lines. The hard part, and the part this article is about, is deciding which errors deserve a second attempt.

It sits in the error handling guide and feeds the full treatment of delays in retry with backoff. Everything ran on Go 1.25.6.

What makes an error retryable

An error is worth retrying when the next attempt can reasonably have a different result, and the failure is not your fault. Both halves matter.

Transient causes are the retryable ones: a network blip, a server restarting, a deadline missed because of load, a rate limit that will reset, a database failover, a lock wait timeout. Time passes and the situation changes.

Permanent causes are not: a malformed request, a missing record, a failed authentication, a validation error, a constraint violation. Trying again sends the same bytes into the same rule, and the answer is the same.

Caller-side stops are a third group that people forget. If the context is cancelled or its deadline passed, the caller no longer wants the result. The failure is "retryable" in the sense that another attempt might succeed and not worth making, so the loop must stop.

For HTTP, the statuses split cleanly enough to put in a table.

A 500 is the grey area. It might be a transient fault or a bug triggered by your request, and retrying a bug gives you the same bug. Allow one or two attempts at most.

Classify by behavior, not by type

The retry loop should not import every package whose errors it might see. Give it a question to ask instead. A small interface that errors can implement keeps the knowledge with the code that has it:

package main

import (
	"context"
	"errors"
	"net"
	"time"
)

type retryable interface{ Retryable() bool }

type permanentError struct{ err error }

func (p permanentError) Error() string { return p.err.Error() }
func (p permanentError) Unwrap() error { return p.err }

// Permanent marks err as one that must not be retried.
func Permanent(err error) error {
	if err == nil {
		return nil
	}
	return permanentError{err}
}

// IsRetryable decides whether another attempt is worth making.
func IsRetryable(err error) bool {
	if err == nil {
		return false
	}
	var perm permanentError
	if errors.As(err, &perm) {
		return false
	}
	if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
		return false
	}
	var r retryable
	if errors.As(err, &r) {
		return r.Retryable()
	}
	var ne net.Error
	if errors.As(err, &ne) {
		return ne.Timeout()
	}
	return false
}

// Retry calls fn up to attempts times, sleeping with doubling delays between tries.
func Retry(ctx context.Context, attempts int, base time.Duration, fn func(context.Context) error) error {
	var err error
	delay := base
	for i := 1; i <= attempts; i++ {
		if err = fn(ctx); err == nil || !IsRetryable(err) {
			return err
		}
		if i == attempts {
			break
		}
		select {
		case <-time.After(delay):
			delay *= 2
		case <-ctx.Done():
			return ctx.Err()
		}
	}
	return err
}

IsRetryable checks in an order that matters. A value explicitly marked permanent wins over everything, then the context errors stop the loop, then an error that says what it is gets believed, then network errors are retried only for timeouts. The last line is the default, and it is false. A loop that retries unknown errors by default makes every new kind of failure somebody's retry storm, and a loop that stops by default makes a new transient error fail once, visibly, which is the cheaper mistake.

The Permanent wrapper keeps the cause reachable through Unwrap, so errors.Is still sees what was wrong. A caller that detected something unrecoverable, such as a 403 that came back in a 200-wrapped body, can mark it and be sure no outer loop will retry.

Using it

package main

import (
	"context"
	"errors"
	"fmt"
	"time"
)

type httpStatusError struct{ Code int }

func (e httpStatusError) Error() string { return fmt.Sprint("status ", e.Code) }
func (e httpStatusError) Retryable() bool {
	switch e.Code {
	case 408, 429, 502, 503, 504:
		return true
	}
	return false
}

func run(name string, ctx context.Context, outcomes ...error) {
	calls := 0
	err := Retry(ctx, 4, time.Millisecond, func(context.Context) error {
		defer func() { calls++ }()
		if calls < len(outcomes) {
			return outcomes[calls]
		}
		return nil
	})
	fmt.Printf("%-18s calls=%d err=%v\n", name, calls, err)
}

func main() {
	bg := context.Background()
	run("transient then ok", bg, httpStatusError{503}, httpStatusError{503})
	run("permanent 404", bg, httpStatusError{404}, nil)
	run("marked permanent", bg, Permanent(errors.New("token revoked")), nil)
	run("always 503", bg, httpStatusError{503}, httpStatusError{503}, httpStatusError{503}, httpStatusError{503})

	ctx, cancel := context.WithCancel(bg)
	cancel()
	run("cancelled context", ctx, context.Canceled, nil)
}

transient then ok  calls=3 err=<nil>
permanent 404      calls=1 err=status 404
marked permanent   calls=1 err=token revoked
always 503         calls=4 err=status 503
cancelled context  calls=1 err=context canceled

The first run fails twice with 503 and succeeds on the third call. The 404 stops after one call, and so does the marked error. The always-failing case gives up after four attempts with the last error, which is what the caller needs to log. The cancelled context never retries. Each line of the output is one rule from the table above.

Don't retry what you can't repeat

A retry is a duplicate request, and the server may have processed the first one. A GET is safe. A PUT or DELETE is idempotent by definition of HTTP, so repeating it is harmless. A POST that charges a card is not: if the response was lost after the charge succeeded, a retry charges twice. The remedy is an idempotency key. The client generates a unique key per logical operation and sends it with each attempt, and the server remembers the key and returns the original result for repeats. Many payment and messaging APIs support this, and if yours does not, restrict retries to failures that happen before the request is sent, such as a refused connection. Timeouts after sending are the ambiguous ones; for a non-idempotent call, treat them as permanent or verify state before repeating.

Test the classification

The classification function is where the bugs hide, and it is trivial to test because it is a pure function of an error. A table with one row per rule documents the policy:

package main

import (
	"context"
	"errors"
	"fmt"
	"testing"
)

func TestIsRetryable(t *testing.T) {
	tests := []struct {
		name string
		err  error
		want bool
	}{
		{"nil", nil, false},
		{"503", httpStatusError{503}, true},
		{"404", httpStatusError{404}, false},
		{"wrapped 503", fmt.Errorf("call: %w", httpStatusError{503}), true},
		{"permanent beats retryable", Permanent(httpStatusError{503}), false},
		{"cancelled", fmt.Errorf("call: %w", context.Canceled), false},
		{"deadline", context.DeadlineExceeded, false},
		{"unknown", errors.New("mystery"), false},
	}
	for _, tc := range tests {
		t.Run(tc.name, func(t *testing.T) {
			if got := IsRetryable(tc.err); got != tc.want {
				t.Errorf("IsRetryable(%v) = %v, want %v", tc.err, got, tc.want)
			}
		})
	}
}

=== RUN   TestIsRetryable
=== RUN   TestIsRetryable/nil
=== RUN   TestIsRetryable/503
=== RUN   TestIsRetryable/404
=== RUN   TestIsRetryable/wrapped_503
=== RUN   TestIsRetryable/permanent_beats_retryable
=== RUN   TestIsRetryable/cancelled
=== RUN   TestIsRetryable/deadline
=== RUN   TestIsRetryable/unknown
--- PASS: TestIsRetryable (0.00s)
    --- PASS: TestIsRetryable/nil (0.00s)
    --- PASS: TestIsRetryable/503 (0.00s)
    --- PASS: TestIsRetryable/404 (0.00s)
    --- PASS: TestIsRetryable/wrapped_503 (0.00s)
    --- PASS: TestIsRetryable/permanent_beats_retryable (0.00s)
    --- PASS: TestIsRetryable/cancelled (0.00s)
    --- PASS: TestIsRetryable/deadline (0.00s)
    --- PASS: TestIsRetryable/unknown (0.00s)
PASS
ok  	example.test/lab	1.260s

Two rows deserve attention. "wrapped 503" proves the check sees through %w, which is why it uses errors.As. "permanent beats retryable" proves the explicit marker overrides a type that claims to be retryable. If you change the policy later, this table is the specification you change first, and the failing row tells you what the change affects.

Respect the server's hint

A 429 or 503 often carries Retry-After, either seconds or a date. Honor it. Ignoring the hint and retrying at your own rhythm makes you the problem the rate limiter exists to stop. The simplest approach is an error that carries the delay, with the retry loop taking the larger of its own backoff and the hint:

var ra interface{ RetryAfter() time.Duration }
if errors.As(err, &ra) && ra.RetryAfter() > delay {
	delay = ra.RetryAfter()
}

Add jitter, a random fraction of the delay, so that a thousand clients that failed together do not all retry together. Retry with backoff covers the schedule and the jitter, and rate limiting covers the other side of the same conversation.

Where classification lives

Put the knowledge near the source. The HTTP client package knows its status table, the database package knows which driver codes mean a deadlock or a serialization failure, and the message queue client knows its own throttling errors. Each implements Retryable() bool (or returns an error type that does), and the application's retry code stays generic. That arrangement also lets you test the classification without any network: build the error, ask the question, assert the answer. For the stdlib's own conventions, see how net.Error exposes Timeout(), and for the status code semantics, RFC 9110.