Test Retry and Backoff Logic in Go Without Waiting

Retry code is a small loop with several decisions in it: how many attempts, how long to wait between them, which errors to retry, when to stop. A mistake in any of them is expensive in a way that is easy to miss. A missing cap hammers a failing dependency. A missing sleep turns a retry into a busy loop. Retrying a request that was rejected as invalid wastes time and, for a non-idempotent call, may do damage twice.

These are all testable in microseconds if the loop does not call time.Sleep. The fix is to make sleeping a dependency, and then the schedule becomes data. This article tests a retry function that way on Go 1.25.6.

The function

package main

import (
	"context"
	"errors"
	"time"
)

var ErrPermanent = errors.New("permanent")

// Retry calls fn up to attempts times, doubling the wait from base. sleep is injected so tests do not wait.
func Retry(ctx context.Context, attempts int, base time.Duration, sleep func(context.Context, time.Duration) error, fn func() error) error {
	var err error
	wait := base
	for i := 0; i < attempts; i++ {
		if err = fn(); err == nil || errors.Is(err, ErrPermanent) {
			return err
		}
		if i == attempts-1 {
			break
		}
		if serr := sleep(ctx, wait); serr != nil {
			return serr
		}
		wait *= 2
	}
	return err
}

func main() {}

Retry calls fn up to attempts times. After a failure it asks sleep to wait, starting at base and doubling each time, and it stops without sleeping after the last attempt. A nil result or an error wrapping ErrPermanent returns immediately. If sleep returns an error, as it should when the context ends, Retry returns that error.

The sleep parameter has the signature func(context.Context, time.Duration) error. In production it waits on a timer or the context, whichever comes first:

func realSleep(ctx context.Context, d time.Duration) error {
	t := time.NewTimer(d)
	defer t.Stop()
	select {
	case <-t.C:
		return nil
	case <-ctx.Done():
		return ctx.Err()
	}
}

That function is small enough to test once by itself, and everything else uses a fake. Taking the clock as a parameter is the same idea as in testing time dependent code: the function reads time only through something the test supplies.

Four tests

package main

import (
	"context"
	"errors"
	"fmt"
	"testing"
	"time"
)

type recorder struct{ waits []time.Duration }

func (r *recorder) sleep(ctx context.Context, d time.Duration) error {
	r.waits = append(r.waits, d)
	return ctx.Err()
}

func TestBackoffSchedule(t *testing.T) {
	var r recorder
	calls := 0
	err := Retry(t.Context(), 5, 100*time.Millisecond, r.sleep, func() error {
		calls++
		if calls < 4 {
			return errors.New("temporary")
		}
		return nil
	})
	t.Logf("err=%v calls=%d waits=%v", err, calls, r.waits)
}

func TestGivesUpAfterAttempts(t *testing.T) {
	var r recorder
	err := Retry(t.Context(), 3, time.Second, r.sleep, func() error { return errors.New("down") })
	t.Logf("err=%v waits=%v (no sleep after the last attempt)", err, r.waits)
}

func TestPermanentErrorIsNotRetried(t *testing.T) {
	var r recorder
	calls := 0
	err := Retry(t.Context(), 5, time.Second, r.sleep, func() error {
		calls++
		return fmt.Errorf("bad request: %w", ErrPermanent)
	})
	t.Logf("calls=%d waits=%d err=%v", calls, len(r.waits), err)
}

func TestContextEndsTheLoop(t *testing.T) {
	var r recorder
	ctx, cancel := context.WithCancel(t.Context())
	calls := 0
	err := Retry(ctx, 10, time.Second, r.sleep, func() error {
		calls++
		if calls == 2 {
			cancel()
		}
		return errors.New("temporary")
	})
	t.Logf("calls=%d err=%v", calls, err)
}

go test -count=1 -v . 2>&1 | grep -vE "^(=== RUN|=== PAUSE|=== CONT|PASS|ok)" | sed -E "s/ \([0-9.]+s\)//; s#^ +[a-z_]+\.go:[0-9]+: #    #"

    err=<nil> calls=4 waits=[100ms 200ms 400ms]
--- PASS: TestBackoffSchedule
    err=down waits=[1s 2s] (no sleep after the last attempt)
--- PASS: TestGivesUpAfterAttempts
    calls=1 waits=0 err=bad request: permanent
--- PASS: TestPermanentErrorIsNotRetried
    calls=2 err=context canceled
--- PASS: TestContextEndsTheLoop

The recorder implements the sleep. It appends each requested wait to a slice and returns ctx.Err(), so a cancelled context ends the loop, the same as the real one. It never waits.

The schedule. TestBackoffSchedule fails the first three calls and succeeds on the fourth. The recorded result is err=<nil> calls=4 waits=[100ms 200ms 400ms]. This is an exact assertion on the backoff: three waits, doubling from the base. A test with real sleeps would take 700 ms and could only check that roughly that much time passed. A mistake such as starting from the wrong base, doubling twice or forgetting to double shows up as a different slice.

The limit. TestGivesUpAfterAttempts never succeeds and allows three attempts. The waits are [1s 2s]: two sleeps for three attempts, because there is nothing to wait for after the last one. A common bug is a final sleep before returning the error, which adds a pointless delay to every failure. The returned error is the last one from fn, down, so the caller sees the real cause and not a generic "retries exhausted". If you prefer to wrap it, assert that errors.Is still finds the original.

The permanent error. TestPermanentErrorIsNotRetried returns an error that wraps ErrPermanent with %w. Retry calls fn once and sleeps zero times; the error comes back as bad request: permanent. Classifying errors is the part of retry logic most likely to be wrong in production, because real errors come from libraries. A test should include the real error types you meet: a 400 versus a 503 from net/http, context.DeadlineExceeded, io.EOF, a database unique violation. The article on error testing covers errors.Is and errors.As in tests.

The context. TestContextEndsTheLoop cancels the context during the second call. The recorder's sleep returns ctx.Err(), so Retry returns context canceled with calls=2. Without this check, a retry loop outlives the request that started it, and a shutdown waits for it.

Jitter

Real backoff adds randomness so that many clients that failed together do not retry together. A test cannot assert an exact random delay, so inject the randomness too: a jitter func(time.Duration) time.Duration parameter, or a *rand.Rand with a fixed seed. The test then checks the bounds: with full jitter, each wait is between zero and the exponential cap. A property test is a good fit, with the bounds as the property, as in property based testing.

What a fake sleep does not test

The recorder proves the schedule you asked for. It says nothing about whether those delays are good ones for the dependency you call. Check the delays against the dependency's documented rate limits and recovery time, and cap the maximum wait. Check, also, that the whole operation has a deadline: unbounded retries with a capped delay run forever, and the caller's context should be the bound.

Two further hazards need their own tests. A retry around a non-idempotent call can duplicate its effect when the first attempt succeeded but the response was lost. The test is to make the fake fail after performing the action, and assert that the retry does not repeat it, typically through an idempotency key. And retries multiply across layers: three layers that each retry three times turn one failure into 27 calls. That is a design review, not a unit test, but a metric on the number of attempts makes it visible.

Table the classification

The permanent-versus-temporary decision is the part of retry logic that changes most often, so give it its own table. Each row is an error and the expected decision: a 503 response retries, a 400 does not, context.DeadlineExceeded from the caller's own context does not, a connection reset retries, io.ErrUnexpectedEOF on a request that may have been partly processed depends on whether the call is idempotent. Put the rows in a function retryable(err) bool and test it separately from the loop, because the loop test then needs only two cases. When a new error class causes an incident, the fix is a new row and a one-line change.

Also test the limit on total time, not only on attempts. A policy of five attempts with a one minute cap is five minutes of waiting in the worst case, and the caller usually has less. Pass a context with a deadline, record the waits with the fake, and assert that the loop stops when the next wait would pass the deadline instead of sleeping into it. That check prevents the case where a retry outlives the request that wanted it.

Third-party helpers

If the code uses a library for retries, such as cenkalti/backoff or avast/retry-go, the same principle holds. Pass its clock or timer option in tests when it has one, and test your classification of errors, which is the part that is yours. Those libraries are described here, not run. The Go-specific pieces are all in the standard library: context for the deadline and errors for classification. For cancellation details, see context cancellation tests.