Run Goroutines Under a Supervisor in Go Services Safely
Build a small supervisor for long running goroutines in Go: restart on failure or panic, exponential backoff, a restart budget, clean cancellation and tests.
Most Go services run a few goroutines that are supposed to last as long as the process: a consumer reading a queue, a watcher on a config file, a connection to a message broker that reconnects, a metrics flusher. When one of them returns an error, or panics, the question is what the program should do. Leaving it dead is how a service ends up running with half of its features silently off. Restarting it in a tight loop is how a bad dependency turns into a traffic spike. Crashing the whole process is sometimes right and sometimes an overreaction.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
A supervisor is the small piece of code that makes this decision on purpose. The idea comes from Erlang, where it is a core part of the runtime; in Go you write it yourself, and it fits in a page. This article builds one with restart, backoff, a restart budget and a clean shutdown, and tests it. It builds on recovering from panics in goroutines, starting and stopping goroutines and retry with backoff, and belongs to the concurrency patterns cluster. Everything ran on Go 1.25.6.
The policy
Four decisions define a supervisor, and they should be written down before the code.
- What counts as an exit? A return of
nilmeans the function finished its job and should not be restarted. A non-nil error or a panic is a failure. A cancelled context means the whole program is shutting down. - How long to wait before restarting? Immediately restarting a function that fails instantly burns a core and floods the logs. Back off exponentially, from a small base to a cap.
- When has it recovered? If a run lasts longer than some threshold, treat the next failure as new and reset the delay. Otherwise a service that fails once a day would carry a maximum backoff forever.
- When to give up? A function that fails repeatedly is not going to fix itself. Allow some number of restarts within a time window; beyond that, stop and return an error so the caller can escalate: exit the process, flip a health check, page someone.
The code
package main
import (
"context"
"errors"
"fmt"
"runtime/debug"
"time"
)
var ErrRestartBudget = errors.New("too many restarts")
// Spec describes one supervised function and its restart policy.
type Spec struct {
Name string
Run func(ctx context.Context) error // should run until ctx ends or it fails
Backoff time.Duration // first delay; doubles after each quick failure
MaxBackoff time.Duration
Healthy time.Duration // a run at least this long resets the backoff
MaxRestarts int // allowed restarts within Window
Window time.Duration
Logf func(format string, args ...any)
}
// safely runs fn and turns a panic into an error that carries the stack.
func safely(fn func() error) (err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("panic: %v\n%s", r, debug.Stack())
}
}()
return fn()
}
// Supervise runs spec.Run until it returns nil, ctx is cancelled, or the restart budget is spent.
func Supervise(ctx context.Context, spec Spec) error {
delay := spec.Backoff
var restarts []time.Time // when each restart happened, for the sliding window
for attempt := 1; ; attempt++ {
started := time.Now()
spec.Logf("%s: start #%d", spec.Name, attempt)
err := safely(func() error { return spec.Run(ctx) })
if ctx.Err() != nil {
spec.Logf("%s: stopped by shutdown", spec.Name)
return ctx.Err()
}
if err == nil {
spec.Logf("%s: finished", spec.Name)
return nil
}
spec.Logf("%s: failed: %v", spec.Name, firstLine(err))
if time.Since(started) >= spec.Healthy {
delay = spec.Backoff // it ran for a while: this failure is a fresh one
}
now := time.Now()
restarts = append(restarts, now)
for len(restarts) > 0 && now.Sub(restarts[0]) > spec.Window {
restarts = restarts[1:] // forget restarts outside the window
}
if len(restarts) > spec.MaxRestarts {
return fmt.Errorf("%s: %w (%d in %v): last error: %v", spec.Name, ErrRestartBudget, len(restarts), spec.Window, firstLine(err))
}
spec.Logf("%s: restarting in %v", spec.Name, delay)
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
delay = min(delay*2, spec.MaxBackoff)
}
}
func firstLine(err error) string {
s := err.Error()
for i, c := range s {
if c == '\n' {
return s[:i]
}
}
return s
}The loop is the policy in code. safely makes a panic indistinguishable, to the loop, from an error, and keeps the stack in the message. After the function ends, the order of checks matters: cancellation first (the program is stopping, never restart), then clean return (done), then failure. The backoff resets only when the run lasted at least Healthy; otherwise it doubles. The restart times go in a slice that is trimmed to the window, so len(restarts) is the number of restarts in the last Window, and exceeding the budget returns a wrapped sentinel error the caller can test with errors.Is. The wait before restarting also watches the context, so shutdown is not delayed by a long backoff (timeouts and cancellation).
It restarts, and it gives up
The first scenario is a flaky worker: it fails twice (once with an error, once with a panic) and then runs until cancelled. The second is a worker that always fails; the supervisor should stop restarting it:
package main
import (
"context"
"errors"
"fmt"
"sync/atomic"
"time"
)
func main() {
logf := func(format string, args ...any) { fmt.Printf(" "+format+"\n", args...) }
fmt.Println("a flaky worker:")
var runs atomic.Int32
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
done <- Supervise(ctx, Spec{
Name: "consumer", Backoff: 10 * time.Millisecond, MaxBackoff: 80 * time.Millisecond,
Healthy: time.Second, MaxRestarts: 5, Window: time.Minute, Logf: logf,
Run: func(ctx context.Context) error {
switch runs.Add(1) {
case 1:
return errors.New("lost connection to broker")
case 2:
var m map[string]int
m["x"] = 1 // a bug: panics
}
<-ctx.Done() // the third run is healthy and runs until shutdown
return nil
},
})
}()
time.Sleep(150 * time.Millisecond)
cancel() // shutdown
fmt.Println(" supervisor returned:", <-done)
fmt.Println("a worker that always fails:")
err := Supervise(context.Background(), Spec{
Name: "doomed", Backoff: time.Millisecond, MaxBackoff: 4 * time.Millisecond,
Healthy: time.Second, MaxRestarts: 3, Window: time.Minute, Logf: logf,
Run: func(ctx context.Context) error { return errors.New("cannot reach the database") },
})
fmt.Println(" supervisor returned:", err)
fmt.Println(" is a budget error:", errors.Is(err, ErrRestartBudget))
}a flaky worker:
consumer: start #1
consumer: failed: lost connection to broker
consumer: restarting in 10ms
consumer: start #2
consumer: failed: panic: assignment to entry in nil map
consumer: restarting in 20ms
consumer: start #3
consumer: stopped by shutdown
supervisor returned: context canceled
a worker that always fails:
doomed: start #1
doomed: failed: cannot reach the database
doomed: restarting in 1ms
doomed: start #2
doomed: failed: cannot reach the database
doomed: restarting in 2ms
doomed: start #3
doomed: failed: cannot reach the database
doomed: restarting in 4ms
doomed: start #4
doomed: failed: cannot reach the database
supervisor returned: doomed: too many restarts (4 in 1m0s): last error: cannot reach the database
is a budget error: trueIn the first scenario the log shows start, failure, a restart in 10 ms, the panic reported with only its first line (the full stack is in the error for whoever wants it), a restart in 20 ms, and then a healthy third run that ends when the context is cancelled. The second shows the delay growing to the cap and the supervisor returning a budget error after the fourth failure within the window. That return value is the escalation: the caller decides whether to log and continue without the component, flip the readiness endpoint to failing, or exit and let the orchestrator restart the whole process.
What a supervisor should not do
Three temptations are worth naming. Do not make the supervisor swallow errors: it logs each failure and returns the final one, because a component that fails ten times is information for the operator. Do not restart without a delay, even the first time, when the failure is likely to be caused by something external; the first restart waiting for the base delay is cheap insurance. And do not let restarts hide state corruption. A function that panics halfway through updating a shared structure leaves that structure in an unknown state, and restarting the goroutine does not repair it. Supervise goroutines whose state is local to them or rebuilt on start; for shared state a panic is a reason to exit the process (when to recover and when to crash).
Tests
The properties to protect are the ones that are easy to break while editing the loop: it must not restart after a clean return, must not restart after cancellation, must respect the budget, and must leave no goroutine running.
package main
import (
"context"
"errors"
"sync/atomic"
"testing"
"time"
)
func quiet(string, ...any) {}
func spec(run func(context.Context) error) Spec {
return Spec{Name: "t", Run: run, Backoff: time.Millisecond, MaxBackoff: 2 * time.Millisecond,
Healthy: time.Second, MaxRestarts: 3, Window: time.Minute, Logf: quiet}
}
func TestCleanReturnIsNotRestarted(t *testing.T) {
var runs atomic.Int32
err := Supervise(context.Background(), spec(func(context.Context) error { runs.Add(1); return nil }))
if err != nil || runs.Load() != 1 {
t.Fatalf("err=%v runs=%d, want nil and 1", err, runs.Load())
}
}
func TestFailuresAreRestartedUntilTheBudgetIsSpent(t *testing.T) {
var runs atomic.Int32
err := Supervise(context.Background(), spec(func(context.Context) error { runs.Add(1); return errors.New("boom") }))
if !errors.Is(err, ErrRestartBudget) {
t.Fatalf("want ErrRestartBudget, got %v", err)
}
if runs.Load() != 4 { // the first run plus three restarts
t.Fatalf("ran %d times, want 4", runs.Load())
}
}
func TestPanicIsRestartedLikeAnError(t *testing.T) {
var runs atomic.Int32
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
done <- Supervise(ctx, spec(func(ctx context.Context) error {
if runs.Add(1) == 1 {
panic("first run explodes")
}
<-ctx.Done()
return nil
}))
}()
for runs.Load() < 2 {
time.Sleep(time.Millisecond)
}
cancel()
if err := <-done; !errors.Is(err, context.Canceled) {
t.Fatalf("want context.Canceled, got %v", err)
}
}
func TestCancelDuringBackoffReturnsPromptly(t *testing.T) {
s := spec(func(context.Context) error { return errors.New("fail") })
s.Backoff, s.MaxBackoff = time.Hour, time.Hour // a restart would wait an hour
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
defer cancel()
start := time.Now()
err := Supervise(ctx, s)
if !errors.Is(err, context.DeadlineExceeded) || time.Since(start) > time.Second {
t.Fatalf("err=%v after %v: shutdown was held up by the backoff", err, time.Since(start))
}
}=== RUN TestCleanReturnIsNotRestarted
--- PASS: TestCleanReturnIsNotRestarted (0.00s)
=== RUN TestFailuresAreRestartedUntilTheBudgetIsSpent
--- PASS: TestFailuresAreRestartedUntilTheBudgetIsSpent (0.01s)
=== RUN TestPanicIsRestartedLikeAnError
--- PASS: TestPanicIsRestartedLikeAnError (0.00s)
=== RUN TestCancelDuringBackoffReturnsPromptly
--- PASS: TestCancelDuringBackoffReturnsPromptly (0.03s)
PASS
ok example.test/lab 1.306sThe last test is the one that pays for the select in the wait: with an hour of backoff the supervisor must still return as soon as the context ends. The budget test pins the exact count (four runs for a budget of three restarts) so an off-by-one change shows immediately. All four run under -race.
Where it fits in a program
Use the supervisor for goroutines that should live as long as the process and that can recover by starting over: a message consumer that reconnects, a poller, a file watcher. Compose it with the lifecycle manager from starting and stopping goroutines, whose Start launches a supervised goroutine and whose Stop cancels its context and waits for Supervise to return.
Do not supervise request-scoped work. A goroutine that handles one request should fail that request, not restart. Be careful too with restarts that have side effects: a consumer that restarts must be safe to run again, which usually means resuming from a committed offset and processing idempotently. And report supervisor events as metrics, not only log lines: a restart counter per component is the cheapest early warning that something is unhealthy, long before the budget runs out.
For a program with many such components, github.com/oklog/run offers a similar "group of actors" abstraction, and the suture library implements Erlang-style supervision trees; both are worth reading even if you write your own. The errgroup package covers the different case of stopping everything when one goroutine fails (errgroup). The language-level reference for what recover can and cannot do is the specification's section on handling panics, and the Go blog's post on context covers the cancellation used throughout.
