Idempotency Keys in Go APIs for Safe Client Retries
Make POST safe to retry in a Go API with an Idempotency-Key middleware: concurrent duplicates, replayed responses, payload fingerprints and 5xx handling.
A client sends a payment request and the connection drops. It never learns whether the server received it. It cannot tell "the server never saw this" from "the server charged the card and the response got lost", and it has to choose between two bad options: retry and risk a double charge, or give up and risk a lost one. Networks guarantee that this situation arises, and any API that creates something with POST needs an answer for it.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
The answer is the idempotency key: a unique string the client sends with the request, which lets the server recognise a retry and return the original result instead of doing the work again. Payment APIs popularised it, and it applies to anything with a side effect that must happen once: orders, emails, transfers, signups. This article builds the middleware in Go with the standard library, runs it against five simultaneous retries, and covers the failure cases that decide whether it is correct. It extends the cornerstone JSON API and pairs with the client side described in retries and backoff. Everything ran on Go 1.25.6.
What needs a key
HTTP defines some methods as idempotent: sending the same GET, PUT or DELETE several times has the same effect as sending it once (RFC 9110). A client can retry them freely. POST is not, because creating a resource twice produces two resources. PATCH is not guaranteed either.
An idempotency key turns a POST into something retry-safe by contract. The client generates a random value, such as a UUID, once per logical operation, and sends it as an Idempotency-Key header on the first attempt and on every retry. The server promises: for a given key, the operation takes effect at most once, and every attempt gets the same answer. An IETF draft, the Idempotency-Key HTTP header field, describes the header, and several large payment APIs use the same shape.
The middleware
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"io"
"net/http"
"net/http/httptest"
"sync"
)
type entry struct {
fingerprint string
done chan struct{} // closed when the first request finishes
failed bool // the first request ended in a 5xx and the key was released
status int
header http.Header
body []byte
}
// Store remembers one outcome per key. This one lives in memory; production keeps it in the database or Redis with an expiry.
type Store struct {
mu sync.Mutex
entries map[string]*entry
}
func NewStore() *Store { return &Store{entries: map[string]*entry{}} }
// fingerprint ties a key to the request it was first used with, so a key reused for a different payload is caught.
func fingerprint(r *http.Request, body []byte) string {
h := sha256.New()
h.Write([]byte(r.Method + " " + r.URL.Path + "\n"))
h.Write(body)
return hex.EncodeToString(h.Sum(nil))
}
func problem(w http.ResponseWriter, status int, title string) {
w.Header().Set("Content-Type", "application/problem+json")
w.WriteHeader(status)
w.Write([]byte(`{"title":"` + title + `","status":` + itoa(status) + "}\n"))
}
func itoa(n int) string {
return string([]byte{byte('0' + n/100), byte('0' + n/10%10), byte('0' + n%10)})
}
// Middleware makes unsafe methods safe to retry when the client sends an Idempotency-Key.
func (s *Store) Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key := r.Header.Get("Idempotency-Key")
if r.Method != http.MethodPost || key == "" {
next.ServeHTTP(w, r)
return
}
if len(key) > 255 {
problem(w, 400, "Idempotency-Key is too long")
return
}
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
if err != nil {
problem(w, 413, "Body too large")
return
}
r.Body = io.NopCloser(bytes.NewReader(body))
fp := fingerprint(r, body)
s.mu.Lock()
if e, ok := s.entries[key]; ok {
s.mu.Unlock()
if e.fingerprint != fp {
problem(w, 422, "Idempotency-Key was used with a different request")
return
}
select {
case <-e.done:
case <-r.Context().Done():
return
}
if e.failed {
w.Header().Set("Retry-After", "1")
problem(w, 503, "The first attempt failed; retry")
return
}
for k, v := range e.header {
w.Header()[k] = v
}
w.Header().Set("Idempotent-Replayed", "true")
w.WriteHeader(e.status)
w.Write(e.body)
return
}
e := &entry{fingerprint: fp, done: make(chan struct{})}
s.entries[key] = e
s.mu.Unlock()
rec := httptest.NewRecorder()
next.ServeHTTP(rec, r)
e.status, e.header, e.body = rec.Code, rec.Header().Clone(), rec.Body.Bytes()
if e.status >= 500 { // a server failure is not an answer worth replaying
e.failed = true
s.mu.Lock()
delete(s.entries, key)
s.mu.Unlock()
}
close(e.done)
for k, v := range e.header {
w.Header()[k] = v
}
w.WriteHeader(e.status)
w.Write(e.body)
})
}Four decisions are encoded in it.
State per key. The Store maps a key to an entry holding a fingerprint, a done channel and the saved response. The first request for a key creates the entry and runs the handler. Anyone else with the same key finds the entry and waits on done, which is how concurrent duplicates are handled: they do not run in parallel and do not return early with nothing, they block until the first finishes and then receive its result. The wait also honors the request context, so a client that gives up does not leave a goroutine behind.
A fingerprint of the request. The entry remembers a SHA-256 of the method, path and body. If a second request arrives with the same key but a different fingerprint, the client has a bug, perhaps a key reused across two different orders, and replaying the first response would hide it. The middleware returns a 422 with a clear title. A key is bound to one request.
Only successes and client errors are replayed. A 201 or a 400 is an answer the server gave on purpose, and returning the same one on retry is the contract. A 5xx is a failure, possibly before the work was done, so replaying it would pin the client to an error forever. The middleware drops the key on a 5xx and closes done with failed set; any request that was waiting gets a 503 with Retry-After, and the next attempt starts fresh. The point to watch is the one under "Failure windows" below.
Scope. Only POST with a key is intercepted. Everything else passes straight through, including a keyless POST, which stays as unsafe as it was. Whether to require the key on certain endpoints, answering a missing one with a 400, is a policy choice. For anything that moves money, require it.
Five clients at once
The program starts a server with a payments endpoint whose handler sleeps for 50 ms and then increments a counter, so retries can overlap the original request.
package main
import (
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"time"
)
func main() {
var charges, attempts atomic.Int64
failFirst := atomic.Bool{}
mux := http.NewServeMux()
mux.HandleFunc("POST /payments", func(w http.ResponseWriter, r *http.Request) {
attempts.Add(1)
if r.Header.Get("X-Flaky") != "" && !failFirst.Swap(true) {
http.Error(w, "database unavailable", 500)
return
}
time.Sleep(50 * time.Millisecond) // the charge takes time, so retries can overlap it
n := charges.Add(1)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(201)
fmt.Fprintf(w, `{"payment":%d}`+"\n", n)
})
srv := httptest.NewServer(NewStore().Middleware(mux))
defer srv.Close()
post := func(key, body string, hdr ...string) (int, string, string) {
req, _ := http.NewRequest("POST", srv.URL+"/payments", strings.NewReader(body))
if key != "" {
req.Header.Set("Idempotency-Key", key)
}
for i := 0; i+1 < len(hdr); i += 2 {
req.Header.Set(hdr[i], hdr[i+1])
}
res, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
b, _ := io.ReadAll(res.Body)
return res.StatusCode, strings.TrimSpace(string(b)), res.Header.Get("Idempotent-Replayed")
}
fmt.Println("== a client retries the same payment 5 times at once (a double click, a timeout and retry)")
var wg sync.WaitGroup
results := make([]string, 5)
for i := range results {
wg.Add(1)
go func() {
defer wg.Done()
st, b, rep := post("pay-123", `{"amount":500}`)
results[i] = fmt.Sprintf("%d %s replayed=%q", st, b, rep)
}()
}
wg.Wait()
for _, r := range results {
fmt.Println(" ", r)
}
fmt.Println("charges:", charges.Load(), " handler attempts:", attempts.Load())
fmt.Println("\n== same key later, same body: replayed from the store")
st, b, rep := post("pay-123", `{"amount":500}`)
fmt.Printf(" %d %s replayed=%q charges=%d\n", st, b, rep, charges.Load())
fmt.Println("\n== same key, different body")
st, b, _ = post("pay-123", `{"amount":999}`)
fmt.Printf(" %d %s\n", st, b)
fmt.Println("\n== a new key is a new payment")
st, b, _ = post("pay-456", `{"amount":500}`)
fmt.Printf(" %d %s charges=%d\n", st, b, charges.Load())
fmt.Println("\n== the first attempt fails with a 500, the retry with the same key runs again")
st, b, _ = post("pay-789", `{"amount":100}`, "X-Flaky", "1")
fmt.Printf(" first: %d %s\n", st, b)
st, b, rep = post("pay-789", `{"amount":100}`, "X-Flaky", "1")
fmt.Printf(" retry: %d %s replayed=%q charges=%d\n", st, b, rep, charges.Load())
fmt.Println("\n== no key at all: nothing is protected")
post("", `{"amount":1}`)
post("", `{"amount":1}`)
fmt.Println(" charges after two keyless posts:", charges.Load())
}== a client retries the same payment 5 times at once (a double click, a timeout and retry)
201 {"payment":1} replayed=""
201 {"payment":1} replayed="true"
201 {"payment":1} replayed="true"
201 {"payment":1} replayed="true"
201 {"payment":1} replayed="true"
charges: 1 handler attempts: 1
== same key later, same body: replayed from the store
201 {"payment":1} replayed="true" charges=1
== same key, different body
422 {"title":"Idempotency-Key was used with a different request","status":422}
== a new key is a new payment
201 {"payment":2} charges=2
== the first attempt fails with a 500, the retry with the same key runs again
first: 500 database unavailable
retry: 201 {"payment":3} replayed="" charges=3
== no key at all: nothing is protected
charges after two keyless posts: 5Read the output top to bottom.
Five goroutines post the same payment with the key pay-123 at the same time. All five receive 201 with {"payment":1}. One of them ran the handler; the other four show replayed="true". The final line confirms it: one charge, one handler attempt. Without the middleware this would have been five charges, and the user would see five line items on their statement.
Later, the same key with the same body returns the stored response again, still marked as replayed, with no new charge. Retrying a day later works the same way as long as the key has not expired.
The same key with a different body (amount 999 instead of 500) gets a 422. A new key creates a new payment, number two.
Then the failure case. The handler returns a 500 on the first attempt for key pay-789, and the middleware discards the key. The client retries with the same key, the handler runs again and succeeds with payment number three. If the middleware had stored the 500, the client would be stuck with it.
The last block shows what the protection is not. Two requests without a key both created payments, bringing the count to five. The key is opt-in on the client side, so the server-side guarantee covers only clients that send one.
Tests
package main
import (
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
)
func TestConcurrentRetriesRunTheHandlerOnce(t *testing.T) {
var runs atomic.Int64
h := NewStore().Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
runs.Add(1)
w.WriteHeader(201)
}))
var wg sync.WaitGroup
for i := 0; i < 20; i++ {
wg.Add(1)
go func() {
defer wg.Done()
req := httptest.NewRequest("POST", "/p", strings.NewReader("x"))
req.Header.Set("Idempotency-Key", "k")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
if rec.Code != 201 {
t.Errorf("status %d", rec.Code)
}
}()
}
wg.Wait()
if runs.Load() != 1 {
t.Fatalf("handler ran %d times", runs.Load())
}
}
func TestDifferentPayloadWithSameKeyIsRefused(t *testing.T) {
h := NewStore().Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(201) }))
do := func(body string) int {
req := httptest.NewRequest("POST", "/p", strings.NewReader(body))
req.Header.Set("Idempotency-Key", "k")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code
}
if a, b := do("one"), do("two"); a != 201 || b != 422 {
t.Fatalf("got %d then %d", a, b)
}
}
func TestGetsAreNeverStored(t *testing.T) {
var runs atomic.Int64
h := NewStore().Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { runs.Add(1) }))
for i := 0; i < 3; i++ {
req := httptest.NewRequest("GET", "/p", nil)
req.Header.Set("Idempotency-Key", "k")
h.ServeHTTP(httptest.NewRecorder(), req)
}
if runs.Load() != 3 {
t.Fatalf("ran %d times", runs.Load())
}
}=== RUN TestConcurrentRetriesRunTheHandlerOnce
--- PASS: TestConcurrentRetriesRunTheHandlerOnce (0.00s)
=== RUN TestDifferentPayloadWithSameKeyIsRefused
--- PASS: TestDifferentPayloadWithSameKeyIsRefused (0.00s)
=== RUN TestGetsAreNeverStored
--- PASS: TestGetsAreNeverStored (0.00s)
PASS
ok example.test/lab 1.289sThe first test starts twenty goroutines with one key and requires the handler to run exactly once, under the race detector. That test is the one to keep: the mistakes in this area (checking the store and writing to it in two separate lock sections, for example) are races that appear only under concurrency. The second pins the 422. The third confirms that GET with a key is never stored.
Failure windows
The in-memory version is correct for one process and is a teaching tool. Four gaps decide whether the production version is.
Crash between the work and the record. The server charges the card, then dies before saving the response. The retry finds no key and charges again. The only fix is to make the key and the side effect atomic: write the key row and the business rows in the same database transaction, so either both exist or neither does. If the side effect is a call to another system, which cannot join your transaction, pass the key through to it so that it can deduplicate, and see the outbox pattern for making the hand-off reliable.
A 500 after a partial effect. Releasing the key on a 5xx assumes the handler did nothing before failing. If it wrote half of an order and then returned an error, the retry produces a duplicate. Handlers behind this middleware must be atomic, or must be written to clean up, which is another argument for transactions.
Memory and multiple servers. A map in one process does not protect a deployment with several replicas, and it grows forever. Store keys in the database or Redis, scope them per authenticated account so one customer cannot collide with another, and expire them. Twenty-four hours is a common window; pick one longer than your clients' retry horizon. A unique index on (account, key) gives you the race-free "first writer wins" for free, and the loser reads the existing row.
Large responses. Replaying means storing the body. For small JSON that is trivial. For large or streaming responses, store a reference to the created resource and replay a lighter answer such as the status and Location, documenting that the replay carries less.
Client rules
The server half only works if clients behave. Document these:
- Generate the key once per logical operation, before the first attempt, with a random UUID or equivalent. Do not derive it from the payload, or two legitimately identical orders collapse into one.
- Reuse the same key for every retry of that operation, and never for a different one.
- Retry on network errors, timeouts, 429 and 5xx, with exponential backoff and jitter, and honor
Retry-After. - Treat a replayed response as the real result. The
Idempotent-Replayedheader is informational only.
On the Go client side, the net/http package will only replay a request automatically in limited cases, so a retry loop with a stable key is code you own.
Alternatives
When the client can choose the resource's identifier, PUT /orders/{id} is idempotent without a key, as in the CRUD API with a natural key. A unique business constraint (one payment per invoice) in the database turns a duplicate into a conflict error. For queues, deduplicate on a message id. The key is the general-purpose tool for the case where none of those fits, and the common case of "create something, return its id".
