Build a Notification Service in Go With Queues and Retries

Every product eventually sends messages on behalf of users: a welcome email, a password reset, an SMS code, a webhook to a customer's system. The first version calls the provider inside the request handler. It works until the provider is slow, at which point signups are slow, or down, at which point signups fail, or flaky, at which point some users never get their code and nobody knows.

The fix is a queue between the thing that wants to send and the thing that sends. The request records the intent and returns. A separate worker delivers it, retries what failed for reasons that might pass, gives up on what never will, and records what happened. This project builds that core in Go: the data model, the delivery loop, the retry policy and the idempotency rule. It stays in memory so the logic is easy to read and test, and the last sections say what to change for durability. It extends retries and backoff and worker pools, and pairs with webhook design for the outgoing side. It ran on Go 1.25.6.

What the service guarantees

State the guarantees first, because they decide the design.

• At-least-once delivery. A notification is attempted until it succeeds or is declared dead. In a crash between "provider accepted" and "we recorded success" it may be sent twice. Exactly-once delivery across a network is not available; the usual answer is to make duplicates harmless, with provider-side idempotency where it exists and clear wording where it does not.
• No duplicates from the caller's retries. If the caller repeats a request because its connection dropped, the idempotency key makes the service return the original.
• Bounded effort. Each notification has a maximum number of attempts and a backoff cap, so a dead recipient does not occupy the system forever.
• Visible state. A notification is queued, sent or dead, with the attempt count and the last error, so support can answer "did it send?".

The model and the delivery loop

package main

import (
	"context"
	"errors"
	"fmt"
	"sort"
	"sync"
	"time"
)

type State string

const (
	Queued State = "queued"
	Sent   State = "sent"
	Dead   State = "dead" // gave up: either a permanent failure or too many attempts
)

type Notification struct {
	ID       string    `json:"id"`
	Channel  string    `json:"channel"`
	To       string    `json:"to"`
	Body     string    `json:"body"`
	Key      string    `json:"-"` // idempotency key from the caller
	State    State     `json:"state"`
	Attempts int       `json:"attempts"`
	Next     time.Time `json:"-"`
	LastErr  string    `json:"last_error,omitempty"`
}

// Provider delivers one notification. Returning ErrPermanent means retrying cannot help.
type Provider interface {
	Send(ctx context.Context, n Notification) error
}

var ErrPermanent = errors.New("permanent failure")

type Config struct {
	MaxAttempts int
	BaseDelay   time.Duration
	MaxDelay    time.Duration
	Workers     int
}

type Service struct {
	cfg       Config
	providers map[string]Provider
	now       func() time.Time

	mu    sync.Mutex
	items map[string]*Notification
	byKey map[string]string
	seq   int
	log   []string
}

func New(cfg Config, providers map[string]Provider, now func() time.Time) *Service {
	return &Service{cfg: cfg, providers: providers, now: now, items: map[string]*Notification{}, byKey: map[string]string{}}
}

// Enqueue stores the notification and returns at once. The same key returns the original, so a retrying caller cannot send twice.
func (s *Service) Enqueue(n Notification) (Notification, bool, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	if _, ok := s.providers[n.Channel]; !ok {
		return Notification{}, false, fmt.Errorf("unknown channel %q", n.Channel)
	}
	if n.Key != "" {
		if id, ok := s.byKey[n.Key]; ok {
			return *s.items[id], false, nil
		}
	}
	s.seq++
	n.ID = fmt.Sprintf("ntf_%03d", s.seq)
	n.State, n.Next = Queued, s.now()
	s.items[n.ID] = &n
	if n.Key != "" {
		s.byKey[n.Key] = n.ID
	}
	return n, true, nil
}

func (s *Service) Get(id string) (Notification, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	n, ok := s.items[id]
	if !ok {
		return Notification{}, false
	}
	return *n, true
}

func (s *Service) backoff(attempt int) time.Duration {
	d := s.cfg.BaseDelay << (attempt - 1)
	if d > s.cfg.MaxDelay || d <= 0 {
		d = s.cfg.MaxDelay
	}
	return d // production code adds jitter; the lab leaves it out so the schedule is exact
}

// RunDue delivers everything that is due now, using a bounded pool of workers, and schedules retries.
func (s *Service) RunDue(ctx context.Context) {
	s.mu.Lock()
	var due []*Notification
	for _, n := range s.items {
		if n.State == Queued && !n.Next.After(s.now()) {
			due = append(due, n)
		}
	}
	s.mu.Unlock()
	sort.Slice(due, func(i, j int) bool { return due[i].ID < due[j].ID })

	sem := make(chan struct{}, s.cfg.Workers)
	var wg sync.WaitGroup
	for _, n := range due {
		wg.Add(1)
		sem <- struct{}{}
		go func() {
			defer wg.Done()
			defer func() { <-sem }()
			s.deliver(ctx, n)
		}()
	}
	wg.Wait()
}

func (s *Service) deliver(ctx context.Context, n *Notification) {
	s.mu.Lock()
	snapshot := *n
	s.mu.Unlock()

	sendCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()
	err := s.providers[snapshot.Channel].Send(sendCtx, snapshot)

	s.mu.Lock()
	defer s.mu.Unlock()
	n.Attempts++
	stamp := s.now().Format("15:04:05")
	switch {
	case err == nil:
		n.State, n.LastErr = Sent, ""
		s.log = append(s.log, fmt.Sprintf("%s %s attempt %d: sent", stamp, n.ID, n.Attempts))
	case errors.Is(err, ErrPermanent):
		n.State, n.LastErr = Dead, err.Error()
		s.log = append(s.log, fmt.Sprintf("%s %s attempt %d: dead (%v)", stamp, n.ID, n.Attempts, err))
	case n.Attempts >= s.cfg.MaxAttempts:
		n.State, n.LastErr = Dead, err.Error()
		s.log = append(s.log, fmt.Sprintf("%s %s attempt %d: dead after %d attempts (%v)", stamp, n.ID, n.Attempts, n.Attempts, err))
	default:
		n.LastErr = err.Error()
		n.Next = s.now().Add(s.backoff(n.Attempts))
		s.log = append(s.log, fmt.Sprintf("%s %s attempt %d: %v, retry at %s", stamp, n.ID, n.Attempts, err, n.Next.Format("15:04:05")))
	}
}

func (s *Service) Log() []string {
	s.mu.Lock()
	defer s.mu.Unlock()
	out := append([]string(nil), s.log...)
	sort.Strings(out) // lines start with the time and then the id, so this orders by time and keeps runs reproducible
	return out
}

The state machine has three states. Queued carries a Next time: the earliest moment the next attempt may happen. Sent and Dead are terminal. Retrying is just updating Next and leaving the state as Queued, which means the same code path handles the first attempt and every retry.

Enqueue validates the channel, checks the idempotency key, assigns an id and stores the item due immediately. It never talks to a provider, so it is fast and cannot fail because of one.

RunDue collects everything that is queued and due, sorts it for a predictable order, and delivers it with at most Workers in flight. The semaphore channel sem is the bound. A burst of 10,000 notifications does not open 10,000 connections to the email provider, and a slow provider cannot consume unbounded goroutines.

Deliver takes a snapshot under the lock, calls the provider with a two second timeout and no lock held (a provider call under a mutex would serialise everything), then records the outcome under the lock again. Calling the provider without the lock is the single most important concurrency decision in this file.

Classifying errors is the heart of the retry policy:

• nil means sent.
• ErrPermanent (wrapped with %w so errors.Is finds it) means retrying cannot help: a malformed address, a blocked recipient, a 4xx from the provider. The notification goes straight to Dead.
• Anything else is assumed transient: a timeout, a 5xx, a refused connection. It is rescheduled with backoff(attempt) unless the attempt cap has been reached, in which case it goes to Dead with the last error.

Providers are responsible for the classification, because only they know whether a given response is permanent. An email provider's adapter would map 400 and 422 to ErrPermanent and 429, 500, 502, 503 and 504 to transient errors, honouring Retry-After for the 429 case.

The driver, with a fake clock

The service takes now func() time.Time instead of calling time.Now. The lab uses that to run twenty seconds of "time" instantly: it advances a variable by one second, calls RunDue, and repeats. The provider is a fake that fails the first N times for a given recipient, fails permanently for one address and never succeeds for another.

package main

import (
	"context"
	"errors"
	"fmt"
	"sync"
	"time"
)

// flaky fails transiently the first n times for each recipient, then succeeds.
type flaky struct {
	mu       sync.Mutex
	failures map[string]int
	sent     []string
}

func (f *flaky) Send(ctx context.Context, n Notification) error {
	f.mu.Lock()
	defer f.mu.Unlock()
	if n.To == "bad-address" {
		return fmt.Errorf("%w: mailbox does not exist", ErrPermanent)
	}
	if f.failures[n.To] > 0 {
		f.failures[n.To]--
		return errors.New("provider returned 503")
	}
	f.sent = append(f.sent, n.To)
	return nil
}

func main() {
	clock := time.Date(2026, 5, 1, 12, 0, 0, 0, time.UTC)
	now := func() time.Time { return clock }

	email := &flaky{failures: map[string]int{"[email protected]": 2, "[email protected]": 99}}
	svc := New(Config{MaxAttempts: 4, BaseDelay: time.Second, MaxDelay: 30 * time.Second, Workers: 3},
		map[string]Provider{"email": email}, now)

	add := func(to, key string) {
		n, created, err := svc.Enqueue(Notification{Channel: "email", To: to, Body: "Welcome", Key: key})
		fmt.Printf("enqueue %-18s key=%-8q -> id=%s created=%v err=%v\n", to, key, n.ID, created, err)
	}
	add("[email protected]", "welcome-1") // fails twice, then succeeds
	add("[email protected]", "welcome-1") // same key: the original is returned, nothing new is queued
	add("[email protected]", "")          // succeeds at once
	add("bad-address", "")               // permanent failure
	add("[email protected]", "")         // keeps failing until it runs out of attempts
	_, _, err := svc.Enqueue(Notification{Channel: "pager", To: "x"})
	fmt.Println("unknown channel:", err)

	ctx := context.Background()
	for second := 0; second <= 20; second++ {
		svc.RunDue(ctx)
		clock = clock.Add(time.Second)
	}
	fmt.Println()
	for _, l := range svc.Log() {
		fmt.Println(l)
	}
	fmt.Println()
	for _, id := range []string{"ntf_001", "ntf_002", "ntf_003", "ntf_004"} {
		n, _ := svc.Get(id)
		fmt.Printf("%s %-18s state=%-6s attempts=%d last_error=%q\n", n.ID, n.To, n.State, n.Attempts, n.LastErr)
	}
	fmt.Println("emails actually delivered:", email.sent)
}

enqueue [email protected]   key="welcome-1" -> id=ntf_001 created=true err=<nil>
enqueue [email protected]   key="welcome-1" -> id=ntf_001 created=false err=<nil>
enqueue [email protected]   key=""       -> id=ntf_002 created=true err=<nil>
enqueue bad-address        key=""       -> id=ntf_003 created=true err=<nil>
enqueue [email protected]  key=""       -> id=ntf_004 created=true err=<nil>
unknown channel: unknown channel "pager"

12:00:00 ntf_001 attempt 1: provider returned 503, retry at 12:00:01
12:00:00 ntf_002 attempt 1: sent
12:00:00 ntf_003 attempt 1: dead (permanent failure: mailbox does not exist)
12:00:00 ntf_004 attempt 1: provider returned 503, retry at 12:00:01
12:00:01 ntf_001 attempt 2: provider returned 503, retry at 12:00:03
12:00:01 ntf_004 attempt 2: provider returned 503, retry at 12:00:03
12:00:03 ntf_001 attempt 3: sent
12:00:03 ntf_004 attempt 3: provider returned 503, retry at 12:00:07
12:00:07 ntf_004 attempt 4: dead after 4 attempts (provider returned 503)

ntf_001 [email protected]   state=sent   attempts=3 last_error=""
ntf_002 [email protected]   state=sent   attempts=1 last_error=""
ntf_003 bad-address        state=dead   attempts=1 last_error="permanent failure: mailbox does not exist"
ntf_004 [email protected]  state=dead   attempts=4 last_error="provider returned 503"
emails actually delivered: [[email protected] [email protected]]

Reading the output

Enqueue. Five calls create four notifications. The second welcome-1 request returned ntf_001 with created=false: the service recognised the key and returned the original. The sixth call used an unknown channel and was refused at the door with a clear error, not stored and failed later.

The attempt log, in time order:

• At 12:00:00 all four are attempted. ntf_002 (Bob) is sent. ntf_003 is dead after one attempt: the provider said the mailbox does not exist, wrapped as a permanent failure. ntf_001 and ntf_004 return 503 and are rescheduled for 12:00:01.
• At 12:00:01 attempt two fails for both, and both are rescheduled two seconds out, 12:00:03.
• At 12:00:03 ntf_001 succeeds on attempt 3, and ntf_004 fails again and is rescheduled four seconds out, 12:00:07.
• At 12:00:07 ntf_004 makes attempt 4, reaches the cap of 4 and becomes dead.

The delays 1, 2 and 4 seconds come from BaseDelay << (attempt-1), capped at MaxDelay. The final table matches: Ada's notification sent after three attempts with the error cleared, Bob's after one, and two dead with their reasons recorded. The provider's own record of what it delivered contains exactly two addresses.

Run it again and the output is identical. Nothing in it depends on wall-clock time or goroutine scheduling, apart from the order in which concurrent attempts finish, which the log sorts. That reproducibility is the payoff of injecting the clock.

Backoff, jitter and the thundering herd

The lab omits jitter so the schedule is exact. In production, add it. If a provider is down for a minute and ten thousand notifications fail at the same moment, ten thousand retries at the same later moment hit the provider the instant it recovers and may knock it over again. Randomising the delay (a random value up to the computed backoff, or plus or minus 20 percent) spreads them out. Also cap the number of attempts and the total age: a password reset code is worthless after ten minutes, so a notification should carry an expiry and be declared dead when it passes, regardless of attempts left.

Making it durable

An in-memory queue loses everything on restart. To persist, the same model maps onto a table:

CREATE TABLE notifications (
  id          bigserial PRIMARY KEY,
  idem_key    text UNIQUE,
  channel     text NOT NULL,
  recipient   text NOT NULL,
  body        text NOT NULL,
  state       text NOT NULL DEFAULT 'queued',
  attempts    int  NOT NULL DEFAULT 0,
  next_at     timestamptz NOT NULL DEFAULT now(),
  last_error  text
);
CREATE INDEX ON notifications (next_at) WHERE state = 'queued';

Workers claim due rows with SELECT ... WHERE state = 'queued' AND next_at <= now() ORDER BY next_at LIMIT 10 FOR UPDATE SKIP LOCKED, which lets several workers on several machines take distinct rows without blocking each other. They deliver, then update state and next_at in a transaction. The unique idem_key column replaces the in-memory map and makes the idempotency check atomic. A partial index on next_at for queued rows keeps the claim query fast however many sent rows pile up. A crash after claiming but before recording leaves the row in its old state, and it is picked up again later: that is the at-least-once behaviour from the guarantees.

If a database queue is not enough, the same service shape works on Redis streams, NATS JetStream, SQS or Kafka. The queue changes; the state machine, the error classification and the idempotency rule stay the same. Dedicated systems such as Temporal or Asynq add scheduling, visibility and retries as features if you do not want to maintain them.

Putting an HTTP front on it

The endpoints are thin, which is how you want them. A sketch of the create handler shows where the guarantees are enforced:

mux.HandleFunc("POST /v1/notifications", func(w http.ResponseWriter, r *http.Request) {
	var in struct {
		Channel string `json:"channel"`
		To      string `json:"to"`
		Body    string `json:"body"`
	}
	dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<16))
	dec.DisallowUnknownFields()
	if dec.Decode(&in) != nil || in.To == "" || in.Body == "" {
		problem(w, http.StatusUnprocessableEntity, "channel, to and body are required")
		return
	}
	n, created, err := svc.Enqueue(Notification{Channel: in.Channel, To: in.To, Body: in.Body, Key: r.Header.Get("Idempotency-Key")})
	if err != nil {
		problem(w, http.StatusUnprocessableEntity, err.Error())
		return
	}
	status := http.StatusAccepted
	if !created {
		status = http.StatusOK // a repeat of an earlier request
	}
	w.Header().Set("Location", "/v1/notifications/"+n.ID)
	writeJSON(w, status, n)
})

The status codes carry meaning. 202 Accepted says "recorded, not yet delivered", which is the truth. 200 OK on a repeated key tells a careful client that its earlier request had already been accepted. The handler never waits for a provider, so its latency does not depend on theirs.

One decision deserves attention: what the key covers. If a caller reuses a key with a different body, the service should reject it with 422 and not return the original silently, because the caller probably has a bug. Store a hash of the request with the key and compare. The idempotency key article discusses the full contract.

Database queue or message broker

Teams often jump to a broker for this problem. Both fit, and the choice depends on what you already run.

A database queue is the right default when you already operate PostgreSQL or MySQL, the volume is moderate (hundreds to a few thousand notifications a second), and you want the notification and the business change to commit atomically. If signing up a user and queueing the welcome email are in the same transaction, you cannot have one without the other, which removes a whole family of bugs. This is the transactional outbox pattern: write the intent to a table in the same transaction, and let a worker deliver it afterwards.

A broker (NATS JetStream, RabbitMQ, SQS, Kafka) makes sense for high volume, many producers and consumers across services, or when you need fan-out to several independent consumers. You pay with another system to run and a dual-write problem: updating the database and publishing to the broker are two operations that can disagree after a crash. The outbox pattern solves it, with the table acting as the staging area and a relay publishing from it.

Whichever you pick, the properties in this article hold: durable state, a bound on concurrency, error classification, idempotency and a terminal state for work that cannot succeed. Choosing the transport is the easy part.

The edges of the service

The API. POST /v1/notifications returns 202 Accepted with the id and a Location for status, the same pattern as async jobs. It takes an Idempotency-Key header and returns 200 with the original on a repeat. GET /v1/notifications/{id} returns the state. Authenticate callers and scope ids to the caller (IDOR).

Provider adapters. One per channel (SMTP or an email API, an SMS gateway, an HTTP webhook caller). Each maps the provider's responses onto success, permanent and transient, sets its own timeout, and uses an SSRF-safe client when the target is customer supplied (SSRF).

Per-provider rate limits. Providers cap your send rate. A token bucket per provider in front of Send keeps the service inside the limit and turns excess into delay and not 429s (rate limiter).

Templates and preferences. Store a template id and variables, not a rendered body, so you can re-render after a bug fix and keep personal data out of long-lived logs. Check the user's preferences and consent at send time and not at enqueue time, since they may have unsubscribed in between.

Ordering. The lab makes no promise about the order in which different notifications are delivered. If order matters for a recipient (a code followed by a correction), serialise by recipient with a key.

Observability. Count by state, attempt number and channel. Alert on dead notifications, on queue age (the oldest queued item's wait), and on provider error rates. Log the id and provider response codes, never the body (logging).

Shutdown. Stop accepting, let in-flight deliveries finish within a deadline, and leave queued items in the store. Nothing is lost when the service stops, because the state is the queue (graceful shutdown).

Testing

The approach in the lab is the approach for tests: a fake clock, a fake provider whose failures are scripted, and assertions on the log and the final states. Add table tests for the error classification, a test that runs RunDue with 100 notifications and 3 workers and asserts that no more than 3 provider calls were ever in flight, and a test for the expiry rule. Run them with -race, since the delivery loop is the concurrent part. The standard library's context documentation covers the per-attempt timeout used in deliver, and the Google Cloud guide to retry strategies is a readable description of exponential backoff with jitter used by a large provider.