Send Webhooks From Go With Retries and Request Signing
Build the sending side of webhooks in Go: signed deliveries, a retry schedule that honors Retry-After, permanent versus temporary failures and dead endpoints.
Receiving a webhook is a security problem. Sending one is a reliability problem, and the reliability problem is the larger of the two, because it belongs to you. When your system tells a customer's server that an order was paid, you have taken on a promise: the customer will find out, even if their server is down for an hour, even if your own deploy restarts the process halfway through. Everything about a good sender follows from how seriously you take that promise.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
This project builds the delivery engine: a signed request, a retry schedule, a classification of failures, and the rules for giving up. It is the counterpart of the webhook receiver, which defines what the other end checks, and the contract you publish to customers is covered in webhook design. It belongs to the web cluster that ends in a production HTTP server and reuses the retry thinking from retries and backoff for the HTTP client. Everything ran on Go 1.25.6 with the standard library only.
What you are promising
State the delivery guarantee to customers in plain words, and make it the one you can keep: at least once. You will try until the endpoint acknowledges with a 2xx or until you give up after a documented schedule. A customer's server that processes an event, crashes before replying, and gets the same event again must cope, which is why every delivery carries a stable event ID. Exactly once is not available over an unreliable network, and promising it makes a support ticket out of every timeout.
Ordering is the second promise to refuse. If event A fails and is retried after an hour while event B succeeds at once, the customer sees B first. Say so, put a monotonically increasing sequence or a timestamp in the payload so receivers can sort and discard stale updates, and keep per-endpoint ordering optional (a lot of throughput depends on not holding B behind A). The header semantics for retrying are in RFC 9110's Retry-After section.
The delivery engine
package main
import (
"bytes"
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"errors"
"net"
"net/http"
"strconv"
"syscall"
"time"
)
// Sign returns the X-Signature value for a body at a time, in the format the receiver article verifies.
func Sign(secret string, t time.Time, body []byte) string {
ts := strconv.FormatInt(t.Unix(), 10)
h := hmac.New(sha256.New, []byte(secret))
h.Write([]byte(ts + "."))
h.Write(body)
return "t=" + ts + ",v1=" + hex.EncodeToString(h.Sum(nil))
}
type Endpoint struct {
ID string
URL string
Secret string
Failures int // consecutive deliveries that used up every attempt
Disabled bool
}
type Attempt struct {
Number int
Waited time.Duration // the delay before this attempt
Status int
Err string
Retry bool
RetryAfter time.Duration
}
var (
ErrDisabled = errors.New("endpoint is disabled")
ErrGaveUp = errors.New("delivery failed")
)
type Dispatcher struct {
Client *http.Client
Schedule []time.Duration // wait before attempt 2, 3, ...; the first attempt is immediate
Sleep func(ctx context.Context, d time.Duration) error
Now func() time.Time
DisableAfter int // consecutive failed deliveries before an endpoint is switched off
}
func NewDispatcher(timeout time.Duration) *Dispatcher {
return &Dispatcher{
Client: &http.Client{
Timeout: timeout,
// A redirect would send a signed body to a place the customer never registered.
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
},
Schedule: []time.Duration{time.Minute, 5 * time.Minute, 30 * time.Minute, 2 * time.Hour},
Sleep: func(ctx context.Context, d time.Duration) error {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-t.C:
return nil
case <-ctx.Done():
return ctx.Err()
}
},
Now: time.Now,
DisableAfter: 3,
}
}
// Deliver sends one event to one endpoint, retrying as the schedule says. It returns every attempt made.
func (d *Dispatcher) Deliver(ctx context.Context, ep *Endpoint, eventID string, body []byte) ([]Attempt, error) {
if ep.Disabled {
return nil, ErrDisabled
}
var attempts []Attempt
var waited time.Duration
for n := 1; ; n++ {
a := d.once(ctx, ep, eventID, body, n)
a.Waited = waited
attempts = append(attempts, a)
if a.Status >= 200 && a.Status < 300 {
ep.Failures = 0
return attempts, nil
}
if !a.Retry || n > len(d.Schedule) {
break
}
wait := d.Schedule[n-1]
if a.RetryAfter > wait {
wait = a.RetryAfter // the receiver may ask for more time, never for less than our schedule
}
waited = wait
if err := d.Sleep(ctx, wait); err != nil {
return attempts, err
}
}
ep.Failures++
if ep.Failures >= d.DisableAfter {
ep.Disabled = true
}
return attempts, ErrGaveUp
}
func (d *Dispatcher) once(ctx context.Context, ep *Endpoint, eventID string, body []byte, n int) Attempt {
a := Attempt{Number: n}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, ep.URL, bytes.NewReader(body))
if err != nil {
a.Err = "bad endpoint URL"
return a // permanent
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "gohacks-webhooks/1.0")
req.Header.Set("X-Webhook-Id", eventID) // stable across retries: the receiver's de-duplication key
req.Header.Set("X-Webhook-Attempt", strconv.Itoa(n))
req.Header.Set("X-Signature", Sign(ep.Secret, d.Now(), body)) // re-signed per attempt so the timestamp is fresh
res, err := d.Client.Do(req)
if err != nil {
var ne net.Error
switch {
case errors.Is(err, syscall.ECONNREFUSED):
a.Err = "connection refused"
case errors.As(err, &ne) && ne.Timeout():
a.Err = "timeout"
default:
a.Err = "network error"
}
a.Retry = true
return a
}
defer res.Body.Close()
a.Status = res.StatusCode
switch {
case res.StatusCode >= 200 && res.StatusCode < 300:
case res.StatusCode >= 300 && res.StatusCode < 400:
a.Err = "redirects are not followed"
case res.StatusCode == http.StatusRequestTimeout, res.StatusCode == http.StatusTooEarly,
res.StatusCode == http.StatusTooManyRequests, res.StatusCode >= 500:
a.Retry = true
if secs, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil && secs > 0 {
a.RetryAfter = time.Duration(secs) * time.Second
}
}
return a
}Four decisions in there are the ones worth defending.
The signature is recomputed for every attempt. The timestamp is inside the signed data, and the receiver rejects timestamps older than a few minutes. A retry that reused the first attempt's signature, two hours later, would be rejected by every receiver that does the right thing.
The event ID header stays the same across attempts, and X-Webhook-Attempt changes. The first is the receiver's de-duplication key; the second is for humans debugging a delivery log.
Redirects are refused. The default client would follow a 301 and re-send the body to a different URL, with the signature attached, to a host the customer never registered with you. If a customer's endpoint moves, they update the registration. Treating a 3xx as a permanent failure with a clear message tells them to.
The failure classification is the policy. Network errors, 408, 425, 429 and every 5xx are temporary: the receiver may recover, so try again later. Every other 4xx is permanent. A 400 or 404 or 410 will not become a 200 on the third try, and hammering an endpoint that says "gone" a total of five times is rude and fills the customer's logs. Retry-After can lengthen the wait but not shorten the schedule, so a receiver that asks for 90 seconds when the schedule says 60 gets 90, and one that asks for 10 still gets the full 60.
Watching it work
The demo runs four receivers and one disabled-after-failure endpoint. A recording Sleep replaces real waiting, so the output shows the schedule without taking two hours.
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"time"
)
func report(name string, attempts []Attempt, err error) {
fmt.Printf("%s -> %v\n", name, err)
for _, a := range attempts {
status := fmt.Sprint(a.Status)
if a.Err != "" {
status = a.Err
if a.Status != 0 {
status = fmt.Sprintf("%d %s", a.Status, a.Err)
}
}
fmt.Printf(" attempt %d after %-7v %s\n", a.Number, a.Waited, status)
}
}
func main() {
var okCalls, sigOK atomic.Int64
ok := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
okCalls.Add(1)
// A real receiver recomputes the HMAC over "t.body"; here we check the header has the expected shape and the ID is set.
if strings.HasPrefix(r.Header.Get("X-Signature"), "t=") && strings.Contains(r.Header.Get("X-Signature"), ",v1=") &&
r.Header.Get("X-Webhook-Id") == "evt_100" && len(body) > 0 {
sigOK.Add(1)
}
}))
defer ok.Close()
var flakyCalls atomic.Int64
flaky := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch flakyCalls.Add(1) {
case 1:
w.Header().Set("Retry-After", "90")
http.Error(w, "busy", http.StatusServiceUnavailable)
case 2:
w.Header().Set("Retry-After", "10")
http.Error(w, "busy", http.StatusTooManyRequests)
}
}))
defer flaky.Close()
gone := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { http.Error(w, "no such hook", http.StatusGone) }))
defer gone.Close()
moved := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "https://elsewhere.example/hook", http.StatusMovedPermanently)
}))
defer moved.Close()
slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { time.Sleep(400 * time.Millisecond) }))
defer slow.Close()
down := httptest.NewServer(http.NotFoundHandler())
downURL := down.URL
down.Close()
d := NewDispatcher(150 * time.Millisecond)
d.Sleep = func(ctx context.Context, _ time.Duration) error { return ctx.Err() } // do not actually wait
ctx := context.Background()
body := []byte(`{"id":"evt_100","type":"order.paid"}`)
deliver := func(name, url string) *Endpoint {
ep := &Endpoint{ID: name, URL: url, Secret: "whsec_demo"}
attempts, err := d.Deliver(ctx, ep, "evt_100", body)
report(name, attempts, err)
return ep
}
deliver("healthy endpoint", ok.URL)
fmt.Println(" receiver saw a signed request with the event ID:", sigOK.Load() == 1)
deliver("busy then recovers", flaky.URL)
deliver("410 Gone", gone.URL)
deliver("301 redirect", moved.URL)
deliver("slow receiver", slow.URL)
ep := deliver("nothing listening", downURL)
fmt.Println()
fmt.Println("more events to the dead endpoint, until it is switched off:")
for i := 0; i < 3 && !ep.Disabled; i++ {
_, err := d.Deliver(ctx, ep, fmt.Sprint("evt_", 101+i), body)
fmt.Printf(" delivery %d: %v (consecutive failures: %d, disabled: %v)\n", i+2, err, ep.Failures, ep.Disabled)
}
_, err := d.Deliver(ctx, ep, "evt_200", body)
fmt.Println("after being disabled:", err)
}healthy endpoint -> <nil>
attempt 1 after 0s 200
receiver saw a signed request with the event ID: true
busy then recovers -> <nil>
attempt 1 after 0s 503
attempt 2 after 1m30s 429
attempt 3 after 5m0s 200
410 Gone -> delivery failed
attempt 1 after 0s 410
301 redirect -> delivery failed
attempt 1 after 0s 301 redirects are not followed
slow receiver -> delivery failed
attempt 1 after 0s timeout
attempt 2 after 1m0s timeout
attempt 3 after 5m0s timeout
attempt 4 after 30m0s timeout
attempt 5 after 2h0m0s timeout
nothing listening -> delivery failed
attempt 1 after 0s connection refused
attempt 2 after 1m0s connection refused
attempt 3 after 5m0s connection refused
attempt 4 after 30m0s connection refused
attempt 5 after 2h0m0s connection refused
more events to the dead endpoint, until it is switched off:
delivery 2: delivery failed (consecutive failures: 2, disabled: false)
delivery 3: delivery failed (consecutive failures: 3, disabled: true)
after being disabled: endpoint is disabledThe healthy endpoint is the baseline: one attempt, signed, with the event ID. The busy endpoint shows how a retry reads in practice: it answered 503 with Retry-After: 90, so the second attempt waited 90 seconds rather than the schedule's 60, then a 429 with Retry-After: 10 still waited the schedule's 5 minutes, because a short Retry-After cannot shorten the schedule, and the third attempt succeeded. The 410 Gone and the 301 stopped after one attempt each: both are permanent. The slow receiver timed out on all five attempts and the dead one was refused on all five, each with the escalating waits of the schedule.
The last lines are the endpoint lifecycle. Each delivery that exhausts every attempt adds one to Failures, and at three the endpoint is switched off, so later events do not spend workers on it. That is the right default, with two additions that matter in practice: email the customer when it happens, with the last error, because from their side "your webhook was disabled" is a surprise; and let them re-enable it from a dashboard. Resetting Failures on any success means an endpoint that flaps is never switched off by a single bad night.
Fan-out and batching
One business event often goes to many endpoints, and the fan-out is where a sender's load comes from. Create one delivery row per subscribed endpoint at the time the event is recorded, so each has its own attempts, schedule and outcome; sharing a single delivery between endpoints makes one customer's failure everyone's problem. Filter by event type per endpoint at creation, so an endpoint subscribed to order.paid never gets a row for order.created. For endpoints that want high volume, offer batching as an option, one request carrying an array of events with the same signature scheme, but keep single-event delivery as the default, because batching complicates partial failure: if the receiver accepts three of five, a plain 200 cannot say which.
What goes in the payload
The body is a contract you cannot change quietly, so keep it boring and versioned. A shape that has held up:
{
"id": "evt_100",
"type": "order.paid",
"created": "2026-03-01T12:00:00Z",
"api_version": "2026-01",
"data": { "order_id": "ord_8841", "amount": 4200, "currency": "usd" }
}The event ID is the de-duplication key and never changes, even on a manual redelivery. type is a dotted name that receivers can route on, and the set of types is an API: adding one is fine, renaming one is a breaking change. created is when the thing happened, which can differ from when the delivery is attempted, and it gives receivers an ordering. api_version lets the shape evolve without breaking existing consumers. Inside data, send the identifiers and the facts that changed, not a full snapshot of every related object. A thin event that says "order ord_8841 was paid" and lets the receiver fetch the current order from your API has two advantages: the payload stays small and free of data you would regret sending, and a receiver that processes events out of order always reads current state instead of a stale copy.
Cap the size (a few hundred kilobytes at most) and reject larger events at creation, because a single 20 MB payload sent to a hundred endpoints is a self-inflicted outage. Never put secrets, tokens or full personal records in a webhook; they pass through the customer's load balancers, logs and proxies.
Why not retry forever
An unbounded retry is a slow leak. Dead endpoints, typo'd URLs and decommissioned servers would accumulate pending deliveries until the queue is mostly retrying things that will never succeed. The schedule above gives up after about three hours; many providers stretch it to several days with intervals growing to hours. Pick a window that matches how long a customer outage reasonably lasts, publish it, and give customers a way to recover after it: a list of failed deliveries in the dashboard and an API to redeliver an event by ID. Events that exhaust the schedule are not deleted, they move to a failed state and stay visible for as long as you retain delivery logs.
When the endpoint itself is judged dead (the consecutive-failure rule above), stop attempting new events immediately and keep storing them for a bounded period, so that after the customer fixes and re-enables the endpoint you can replay what they missed. That replay is the feature that customers praise, and it is cheap once events are persisted.
Tests that pin the policy
The sender's rules are arithmetic on attempts and delays, which makes them easy to test with an injected clock and sleeper. Three tests carry most of the value: a permanent failure is not retried, Retry-After lengthens but never shortens the wait, and every attempt carries a fresh signature timestamp.
package main
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
)
func newDispatcher(t *testing.T) (*Dispatcher, *[]time.Duration, *time.Time) {
t.Helper()
d := NewDispatcher(time.Second)
var waits []time.Duration
clock := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
d.Now = func() time.Time { return clock }
d.Sleep = func(_ context.Context, w time.Duration) error {
waits = append(waits, w)
clock = clock.Add(w)
return nil
}
return d, &waits, &clock
}
func TestPermanentFailureIsNotRetried(t *testing.T) {
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { calls++; http.Error(w, "no", 400) }))
defer srv.Close()
d, waits, _ := newDispatcher(t)
_, err := d.Deliver(context.Background(), &Endpoint{URL: srv.URL, Secret: "s"}, "evt", []byte(`{}`))
if err != ErrGaveUp || calls != 1 || len(*waits) != 0 {
t.Fatalf("err=%v calls=%d waits=%v: a 400 must be tried exactly once", err, calls, *waits)
}
}
func TestRetryAfterLengthensButNeverShortens(t *testing.T) {
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
switch calls {
case 1:
w.Header().Set("Retry-After", "7200") // two hours, longer than the 1 minute schedule
case 2:
w.Header().Set("Retry-After", "1") // shorter than the 5 minute schedule
default:
return
}
http.Error(w, "later", 503)
}))
defer srv.Close()
d, waits, _ := newDispatcher(t)
if _, err := d.Deliver(context.Background(), &Endpoint{URL: srv.URL, Secret: "s"}, "evt", []byte(`{}`)); err != nil {
t.Fatal(err)
}
if len(*waits) != 2 || (*waits)[0] != 2*time.Hour || (*waits)[1] != 5*time.Minute {
t.Errorf("waits = %v, want [2h 5m]", *waits)
}
}
func TestEveryAttemptIsSignedAtItsOwnTime(t *testing.T) {
var mu sync.Mutex
var stamps, ids []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
defer mu.Unlock()
stamps = append(stamps, strings.SplitN(r.Header.Get("X-Signature"), ",", 2)[0])
ids = append(ids, r.Header.Get("X-Webhook-Id"))
if len(stamps) < 3 {
http.Error(w, "again", 502)
}
}))
defer srv.Close()
d, _, _ := newDispatcher(t)
if _, err := d.Deliver(context.Background(), &Endpoint{URL: srv.URL, Secret: "s"}, "evt_7", []byte(`{}`)); err != nil {
t.Fatal(err)
}
if stamps[0] == stamps[1] || stamps[1] == stamps[2] {
t.Errorf("timestamps %v: a retry reused an old signature, and a strict receiver rejects it", stamps)
}
if ids[0] != "evt_7" || ids[1] != "evt_7" || ids[2] != "evt_7" {
t.Errorf("event IDs changed between attempts: %v", ids)
}
}=== RUN TestPermanentFailureIsNotRetried
--- PASS: TestPermanentFailureIsNotRetried (0.00s)
=== RUN TestRetryAfterLengthensButNeverShortens
--- PASS: TestRetryAfterLengthensButNeverShortens (0.00s)
=== RUN TestEveryAttemptIsSignedAtItsOwnTime
--- PASS: TestEveryAttemptIsSignedAtItsOwnTime (0.00s)
PASS
ok example.test/lab 1.366sThe second test reads the policy straight off the table: a two-hour Retry-After produced a two-hour wait, and a one-second one still waited the full five minutes of the schedule. The third catches the bug that only strict receivers notice, a retry that reuses its first signature. Each uses a loopback httptest server, which is the right level here: the behavior under test is the HTTP exchange, including headers, so a mocked RoundTripper would hide the interesting bugs. Run them with -race since the real dispatcher is called from many goroutines.
What to measure
A webhook sender needs its own dashboard, because failures there are quiet: nobody complains until a customer notices that orders stopped appearing. The numbers I watch are the share of deliveries that succeed on the first attempt (a healthy system is above 95 percent, and a drop points at your side or a large customer), the age of the oldest pending delivery (a growing age means workers are stuck or the queue is behind), the count of endpoints disabled per day, and the distribution of attempts to success. Alert on the oldest-pending age, not on error counts, because it measures the thing customers feel. Break each metric down by endpoint, since one customer's bad day should not read as a platform outage.
Log every failure with the endpoint ID and the error category from the attempt (timeout, connection refused, 410), and keep the category names stable so they can be counted. When a large customer reports missing events, the first question is whether you sent them and what they answered, and the delivery log from the previous section answers it in seconds.
Running it in a real system
Persist before you send. An event that exists only in memory is lost when the process restarts. The reliable pattern is the transactional outbox: in the same database transaction that records the business change (order paid), insert the event and one delivery row per subscribed endpoint, with status pending and a next_attempt_at. A worker pool selects due rows with FOR UPDATE SKIP LOCKED, calls Deliver's single-attempt logic, and writes the result back with the next time. The schedule becomes data (attempt and next_attempt_at), the process can crash at any point, and nothing is lost or sent twice because of a restart. The version above keeps a whole retry sequence in one function for clarity; in the persisted version each attempt is its own step.
Bound concurrency per endpoint. One customer with a slow server must not use every worker. Give each endpoint a small limit (two to four concurrent deliveries) and a timeout of five to ten seconds, so a backlog for one customer waits in the queue and not in your goroutines. The primitive is a semaphore per endpoint, as in semaphores in Go.
Keep a delivery log customers can read. For each attempt store the time, URL, status, a clipped response body and the duration. Most support tickets about webhooks are answered by showing the customer the response their server returned, and a "redeliver" button turns a thirty minute debugging session into one click.
Treat the URL as hostile. Customers type the endpoint, and some will type http://169.254.169.254/ or an internal service. Apply the same dial-time guard as in the link preview service, require https in production, and keep the sender in a network segment with no access to your internal systems. Disallow redirects as above.
Rotate secrets without breaking receivers. Support two active secrets per endpoint during rotation and send both signatures (v1=...,v1=...), which the receiver in the previous article accepts. Show a secret to the customer once, at creation, and store only what you need to sign.
Document the contract. The schedule, the timeout, the success codes, the headers, the signing format, the IP ranges you send from, and the guarantee (at least once, unordered). A short, accurate page prevents more incidents than any code in this article, and a good model is the layout of GitHub's webhook documentation.
