The Outbox Pattern in Go for Reliable Event Delivery

A service that changes its database and then tells the rest of the world about it has a problem it cannot solve with careful coding: two systems, one write each, and no transaction that spans both. If the process dies between them, or the broker is down at the wrong moment, the database says an order exists and nobody downstream ever hears. The outbox pattern is the standard fix, and it needs nothing except the database you already have. This guide builds it in Go 1.25.6 against PostgreSQL 18.6, breaks it on purpose in three ways to see what it guarantees, and covers the design decisions around the relay.

It belongs to the database cornerstone and builds on unit of work and transaction boundaries. The relay's locking technique is the one used in a job queue on Postgres.

The dual write problem

The naive version is two lines:

1. Commit the order to the database.
2. Publish order.placed to the broker.

Reverse them and you have the opposite bug: the event goes out and the commit fails, so subscribers act on an order that does not exist. Either order loses atomicity, because the database and the broker are separate systems with separate failure modes. Retrying helps with a transient broker error and does nothing when the process itself dies between the two statements. Distributed transactions across a database and a broker (two phase commit) exist in theory and are unavailable or unwelcome in most real setups.

The idea

Make the event part of the database transaction. Create an outbox table. When the service changes its data, it inserts a row describing the event into that table in the same transaction. Either both commit or neither does. A second process, the relay, reads pending rows, publishes them to the broker, and marks them as sent.

The atomicity problem has moved from "database and broker" to "database and the same database", which is solved. What remains is delivery from the table to the broker, and it can be retried indefinitely because the row stays until the broker has accepted it.

The code

package main

import (
	"context"
	"database/sql"
	"fmt"
	"log"
	"sort"
	"sync"
	"time"

	_ "github.com/jackc/pgx/v5/stdlib"
)

const dsn = "postgres://postgres:[email protected]:55432/lab?sslmode=disable"

// Broker stands in for Kafka or NATS: it records what was published and can be told to be down.
type Broker struct {
	mu   sync.Mutex
	got  []int64
	down bool
}

func (b *Broker) Publish(id int64) error {
	b.mu.Lock()
	defer b.mu.Unlock()
	if b.down {
		return fmt.Errorf("broker unavailable")
	}
	b.got = append(b.got, id)
	return nil
}

func (b *Broker) stats() (delivered, distinct int) {
	b.mu.Lock()
	defer b.mu.Unlock()
	seen := map[int64]bool{}
	for _, id := range b.got {
		seen[id] = true
	}
	return len(b.got), len(seen)
}

func counts(ctx context.Context, db *sql.DB) (orders, events, pending int) {
	_ = db.QueryRowContext(ctx, "select count(*) from orders609").Scan(&orders)
	_ = db.QueryRowContext(ctx, "select count(*) from outbox609").Scan(&events)
	_ = db.QueryRowContext(ctx, "select count(*) from outbox609 where sent_at is null").Scan(&pending)
	return
}

// placeOrder writes the order and its event in one transaction: both exist or neither does.
func placeOrder(ctx context.Context, db *sql.DB, item string, failAfterEvent bool) error {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer tx.Rollback()
	var id int64
	if err := tx.QueryRowContext(ctx, "insert into orders609 (item) values ($1) returning id", item).Scan(&id); err != nil {
		return err
	}
	if _, err := tx.ExecContext(ctx, "insert into outbox609 (topic, payload) values ('order.placed', jsonb_build_object('order_id', $1::bigint))", id); err != nil {
		return err
	}
	if failAfterEvent {
		return fmt.Errorf("simulated failure after both inserts")
	}
	return tx.Commit()
}

// relayOnce sends up to n pending events. crashBeforeCommit models a process dying after the publish.
func relayOnce(ctx context.Context, db *sql.DB, b *Broker, n int, crashBeforeCommit bool) (int, error) {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return 0, err
	}
	defer tx.Rollback()
	rows, err := tx.QueryContext(ctx, "select id from outbox609 where sent_at is null order by id limit $1 for update skip locked", n)
	if err != nil {
		return 0, err
	}
	var ids []int64
	for rows.Next() {
		var id int64
		if err := rows.Scan(&id); err != nil {
			return 0, err
		}
		ids = append(ids, id)
	}
	rows.Close()
	sent := 0
	for _, id := range ids {
		if err := b.Publish(id); err != nil {
			break
		}
		sent++
		if _, err := tx.ExecContext(ctx, "update outbox609 set sent_at = now() where id = $1", id); err != nil {
			return sent, err
		}
	}
	if crashBeforeCommit {
		return sent, fmt.Errorf("crashed after publishing %d events and before commit", sent)
	}
	return sent, tx.Commit()
}

func main() {
	ctx := context.Background()
	db, err := sql.Open("pgx", dsn)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()
	db.SetMaxOpenConns(8)
	for _, q := range []string{
		"drop table if exists outbox609", "drop table if exists orders609",
		"create table orders609 (id bigserial primary key, item text not null)",
		"create table outbox609 (id bigserial primary key, topic text not null, payload jsonb not null, sent_at timestamptz)",
	} {
		if _, err := db.ExecContext(ctx, q); err != nil {
			log.Fatal(err)
		}
	}
	line := func(label string) {
		o, e, p := counts(ctx, db)
		fmt.Printf("%-48s orders=%d events=%d pending=%d\n", label, o, e, p)
	}

	// The dual write: commit the order, then publish. The broker is down.
	b := &Broker{down: true}
	var id int64
	_ = db.QueryRowContext(ctx, "insert into orders609 (item) values ('dual write') returning id").Scan(&id)
	err = b.Publish(id)
	fmt.Println("dual write publish error:", err)
	line("1. order committed, publish failed:")

	// The outbox: the event is part of the same transaction.
	fmt.Println("rollback case:", placeOrder(ctx, db, "rolled back", true))
	line("2. failed transaction leaves neither row:")
	for i := 1; i <= 20; i++ {
		if err := placeOrder(ctx, db, fmt.Sprintf("item %d", i), false); err != nil {
			log.Fatal(err)
		}
	}
	line("3. twenty orders placed through the outbox:")

	// Relay while the broker is still down: nothing is lost, nothing is marked sent.
	sent, _ := relayOnce(ctx, db, b, 50, false)
	fmt.Println("relay with the broker down sent:", sent)
	line("4. after a relay pass against a dead broker:")

	// Broker back. Two relays run at once; skip locked keeps them off each other's rows.
	b.down = false
	var wg sync.WaitGroup
	for i := 0; i < 2; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for {
				n, err := relayOnce(ctx, db, b, 5, false)
				if err != nil || n == 0 {
					return
				}
				time.Sleep(5 * time.Millisecond)
			}
		}()
	}
	wg.Wait()
	d, u := b.stats()
	line("5. two relays drained the outbox:")
	fmt.Printf("   delivered %d, distinct %d\n", d, u)

	// A relay that dies after publishing and before it commits: at-least-once delivery.
	_ = placeOrder(ctx, db, "late order", false)
	_, err = relayOnce(ctx, db, b, 5, true)
	fmt.Println("crash case:", err)
	line("6. after the crash, before the next relay:")
	_, _ = relayOnce(ctx, db, b, 5, false)
	d, u = b.stats()
	line("7. after the next relay:")
	fmt.Printf("   delivered %d, distinct %d\n", d, u)
	var dup []int64
	seen := map[int64]int{}
	for _, id := range b.got {
		seen[id]++
		if seen[id] == 2 {
			dup = append(dup, id)
		}
	}
	sort.Slice(dup, func(i, j int) bool { return dup[i] < dup[j] })
	fmt.Println("   delivered twice:", dup)
}

dual write publish error: broker unavailable
1. order committed, publish failed:              orders=1 events=0 pending=0
rollback case: simulated failure after both inserts
2. failed transaction leaves neither row:        orders=1 events=0 pending=0
3. twenty orders placed through the outbox:      orders=21 events=20 pending=20
relay with the broker down sent: 0
4. after a relay pass against a dead broker:     orders=21 events=20 pending=20
5. two relays drained the outbox:                orders=21 events=20 pending=0
   delivered 20, distinct 20
crash case: crashed after publishing 1 events and before commit
6. after the crash, before the next relay:       orders=22 events=21 pending=1
7. after the next relay:                         orders=22 events=21 pending=0
   delivered 22, distinct 21
   delivered twice: [22]

The program has three parts. placeOrder is the write side: it opens a transaction, inserts the order, inserts an outbox row with a JSON payload carrying the order id, and commits. relayOnce is the relay: in one transaction it selects up to n unsent rows for update skip locked, publishes each to the Broker, marks each sent_at, and commits. The Broker is a stand-in with a switch that makes it fail, and a record of everything delivered so duplicates can be counted. main runs seven steps, and each line prints three counts: orders, events and pending events.

What each step shows

1. The dual write. The order was inserted and committed, then publishing failed with broker unavailable. Result: orders=1 events=0. The order exists, no event exists, and nothing records that one is owed. This is the state the pattern removes.
2. A failed transaction. placeOrder was told to fail after both inserts. Result: the counts did not change, still orders=1 events=0. Both rows were rolled back together. An outbox row cannot exist without its order.
3. Twenty orders through the outbox. Result: orders=21 events=20 pending=20. Every committed order has exactly one event waiting, and the one from step 1 has none, because it never used the outbox.
4. A relay pass against a dead broker. It sent 0, and pending=20 stayed. The relay's publish failed, so it did not mark anything. Nothing is lost during an outage; the table is the buffer.
5. Two relays, broker back. Both ran in loops taking five rows at a time. The result was pending=0, delivered 20, distinct 20. for update skip locked made each relay skip rows the other had locked, so no event went to the broker twice, and the work was shared.
6. A crash after publish, before commit. The relay published one event and then "died" before committing, so its sent_at update was rolled back. Result: orders=22 events=21 pending=1. The broker has the event and the table still says pending.
7. The next relay. It published the same row again. Result: delivered 22, distinct 21, with event 22 delivered twice. That is the cost of the pattern.

At least once, never exactly once

Step 7 is not a bug in this code. It is an unavoidable property of the design. Between publishing to the broker and committing the sent_at update there is a window, and a crash inside it means the event was sent and not recorded. The relay cannot tell, after restart, that the broker already has it. The options are to risk loss (mark sent first, then publish) or risk duplicates (publish first, then mark). Reliable systems choose duplicates, and the pattern therefore guarantees at least once delivery.

That moves a requirement onto consumers: handling an event twice must be harmless. The practical ways:

• A natural idempotency key. Include the outbox row's id (or the order id plus event type) in the message. The consumer records ids it has processed, in a table with a unique constraint, in the same transaction as its own effect. A second delivery fails the insert and is skipped.
• Idempotent operations. "Set the order's status to shipped" is safe to repeat. "Add one to a counter" is not, unless it is keyed.
• Broker features. Some brokers offer deduplication windows or idempotent producers. Treat them as a mitigation, not a guarantee, because the window is finite.

Ordering needs a decision too. Rows are relayed in id order, but with two relays and a broker that does not preserve order across partitions, event 7 can arrive before event 6. If order matters per entity, partition the broker by the entity id and make sure all events for that entity pass through one relay, or number events per entity and let the consumer reject gaps.

Designing the table

A good outbox row carries everything the consumer needs without a join back:

Add a partial index on the pending rows, create index on outbox (id) where sent_at is null, so the relay's query stays fast however many sent rows pile up. Without it, the relay scans a growing table to find the few unsent rows.

Build the payload from data the transaction already holds, and put the event version in the body. An event is a contract with other teams, and it outlives the schema of the tables it came from.

Running the relay

The relay is a loop that selects a batch, publishes, marks and commits. Several decisions shape it:

• Batch size. Larger batches mean fewer round trips and a larger duplicate window after a crash. A few dozen to a few hundred is typical.
• Hold the transaction through the publish, or not? The lab holds it, which keeps the row locked while the broker call happens and makes skip locked work without an extra state. The cost is a database transaction open during a network call. If the broker is slow, many connections are held. Set a timeout on the publish call from the context, as in query timeouts, and keep batches small.
• Polling or notification. A relay that polls every 200ms is simple and adds up to 200ms of latency. To cut that, the service can send a NOTIFY after commit and the relay LISTENs, falling back to a slower poll as a safety net; LISTEN and NOTIFY from Go shows the mechanics. Notifications can be lost, so never rely on them alone.
• Retries and poison messages. A row the broker rejects forever blocks nothing here, because the relay moves on, but it stays pending and gets retried every pass. Count attempts, back off, and move a row to a dead letter state after a limit so a person looks at it.
• Many instances. Run as many relays as you like. skip locked shares the rows, as step 5 showed, and the cost is losing strict global ordering.

Cleaning up

Sent rows are history that grows without bound. Delete them in batches by sent_at after a retention period, or partition the table by day and drop partitions. Keep a few days: they are your record when a consumer says an event never arrived. Monitor two numbers, the count of pending rows and the age of the oldest one. A rising age means the relay is down, stuck or slower than the writers, and it is the alert that matters.

The alternative: change data capture

Instead of a relay polling the table, a CDC tool can read PostgreSQL's write ahead log through logical replication and publish each insert into the outbox table. Debezium is the best known; in Go you can build a small consumer with a replication connection, as described in a logical replication client with pgx. CDC removes polling latency and the load of the polling query, and it takes on an operational component to run, a replication slot to watch (an abandoned slot makes the server keep WAL until the disk fills) and more to learn. Start with the polling relay. Move to CDC when the polling cost or latency is measurably a problem.

The consumer side: an inbox

Duplicates are the consumer's problem, and the cleanest answer mirrors the outbox. The consumer keeps an inbox table with a unique column for the event id. For each message it opens a transaction, inserts the id with on conflict do nothing, and checks the affected row count. If one row was inserted, it performs its own effect in the same transaction and commits. If none was inserted, the message is a duplicate: commit and acknowledge without doing anything. The effect and the record of having handled the event commit together, so a crash between them is impossible, which gives effectively once processing out of at least once delivery.

Keep inbox rows for at least as long as the broker can redeliver, and for the outbox retention period, so a replay from the table after an incident is still recognised. If the consumer's effect is naturally idempotent, the inbox can be skipped. If it sends an email or charges a card, it cannot.

Testing the pattern

Three tests give most of the confidence, and all of them are cheap with a real database and a fake broker, as in the lab:

1. Atomicity. Force a failure after the outbox insert and assert that neither row exists. This is step 2 above, and it fails if someone moves the insert outside the transaction.
2. Outage. Make the broker fail, run the relay, assert that nothing was marked sent, bring the broker back and assert everything arrives.
3. Duplicates. Kill the relay between publish and commit, run it again, and assert that the consumer's visible result is the same as if the event had arrived once.

Test the relay against PostgreSQL itself, not a mock, because the behaviour under test is locking: skip locked and transaction rollback do not exist in a fake. The setup from testing against PostgreSQL is enough. Also run two relays in the same test with the race detector on, since a bug that lets two take the same row appears only under concurrency.

When you do not need it

If the downstream effect can tolerate being lost or can be recomputed, such as a cache invalidation that also expires on its own, a plain publish after commit is simpler and good enough. If both the data and the consumer live in the same database, a transaction already covers it, and a trigger or a queue table consumed in the same transaction replaces the broker. The outbox is for the case that is common in service architectures: a database change that must reliably lead to a message to another system.

The pattern is described by Chris Richardson on microservices.io, and the locking clause the relay uses is documented in the PostgreSQL manual under the locking clause.