Build a Small API Gateway in Go With Plugin Style Filters

The previous article argued that you should usually configure a gateway rather than write one, and named the case where writing one is right: a small, specific set of behaviours that you want in your own language, tests and release process. This is that case worked through. The result is under two hundred lines of Go and does the jobs a gateway has to do: route by path prefix, authenticate callers, limit them, forward identity safely, balance across upstreams and fail clearly.

The design idea worth taking away even if you never ship this: a gateway is a route table plus filters. Each route names its upstreams and an ordered list of filters, and the filter list is the entire policy for that route. Adding a new behaviour means writing one function. It ran on Go 1.25.6, and uses golang.org/x/time/rate for the limiter. The ground rules for the proxy piece are in the reverse proxy article, and the middleware shape is the one from HTTP middleware in Go.

The shape of the code

Three files, each with one job:

• filters.go: the Filter type and five filters.
• gateway.go: the route table, route matching, the reverse proxy and failover.
• main.go: a demonstration with real backends and a set of requests.

A Filter is func(http.Handler) http.Handler, the standard middleware signature, so every filter you already have works unchanged and the chain composes with one small function. Putting the chain together once at start-up, instead of per request, keeps the hot path allocation-free apart from the proxy itself.

The filters

package main

import (
	"fmt"
	"net/http"
	"sync"
	"sync/atomic"

	"golang.org/x/time/rate"
)

// A Filter wraps the next handler. The chain for a route is built once, at start-up.
type Filter func(http.Handler) http.Handler

func chain(h http.Handler, filters ...Filter) http.Handler {
	for i := len(filters) - 1; i >= 0; i-- {
		h = filters[i](h)
	}
	return h
}

// RequestID gives every request an id, unless one arrived from a trusted caller.
func RequestID() Filter {
	var n atomic.Int64
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			id := fmt.Sprintf("req-%03d", n.Add(1))
			r.Header.Set("X-Request-Id", id)
			w.Header().Set("X-Request-Id", id)
			next.ServeHTTP(w, r)
		})
	}
}

// StripIdentity removes headers that only the gateway may set. It must run before anything that trusts them.
func StripIdentity() Filter {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			r.Header.Del("X-User-Id")
			r.Header.Del("X-Scopes")
			next.ServeHTTP(w, r)
		})
	}
}

type Client struct{ ID, Scopes string }

// APIKey looks the key up and passes the identity downstream in headers the services trust.
func APIKey(keys map[string]Client) Filter {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			c, ok := keys[r.Header.Get("X-API-Key")]
			if !ok {
				w.Header().Set("WWW-Authenticate", `ApiKey realm="gateway"`)
				http.Error(w, `{"error":"invalid api key"}`, http.StatusUnauthorized)
				return
			}
			r.Header.Del("X-API-Key") // the upstream never needs the credential
			r.Header.Set("X-User-Id", c.ID)
			r.Header.Set("X-Scopes", c.Scopes)
			next.ServeHTTP(w, r)
		})
	}
}

// RateLimit allows each client a burst and then one request per interval.
func RateLimit(limit rate.Limit, burst int) Filter {
	var mu sync.Mutex
	limiters := map[string]*rate.Limiter{}
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			id := r.Header.Get("X-User-Id")
			mu.Lock()
			l, ok := limiters[id]
			if !ok {
				l = rate.NewLimiter(limit, burst)
				limiters[id] = l
			}
			mu.Unlock()
			if !l.Allow() {
				w.Header().Set("Retry-After", "1")
				http.Error(w, `{"error":"rate limit exceeded"}`, http.StatusTooManyRequests)
				return
			}
			next.ServeHTTP(w, r)
		})
	}
}

// RequireScope is the coarse check a gateway can do. Object-level checks stay in the service.
func RequireScope(scope string) Filter {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			if r.Header.Get("X-Scopes") != scope {
				http.Error(w, `{"error":"insufficient scope"}`, http.StatusForbidden)
				return
			}
			next.ServeHTTP(w, r)
		})
	}
}

Each filter does one thing, and the order in which a route lists them matters.

RequestID stamps the request and the response with an identifier. Here it is a counter so the output is stable; production code uses a random id or accepts one from a trusted upstream proxy, and puts it in logs and traces, as in request ID middleware.

StripIdentity removes X-User-Id and X-Scopes from the incoming request. This is the filter most hand-written gateways forget. The services behind the gateway trust those headers because only the gateway can set them. If a client could send them, a caller would be whoever they say they are. The rule: delete first, set later.

APIKey looks up the key, answers 401 with a WWW-Authenticate challenge on a miss, and on a hit removes the credential and sets the identity headers. Deleting X-API-Key means the upstream cannot log it, leak it in an error page or forward it. In production the lookup is a constant-time comparison of a hash, as described in API key generation, and it is cached.

RateLimit keeps a token bucket per user id. It must come after authentication, because the id is what it keys on. A real deployment uses a shared store such as Redis so every gateway instance applies the same limit; see the Redis rate limiter. The 429 carries Retry-After.

RequireScope is the one authorisation check a gateway should make: does this client have the scope for this route at all? It does not ask whether this user may read this order. That decision needs data the gateway does not have, and it belongs in the service.

The gateway and the proxy

package main

import (
	"errors"
	"net"
	"net/http"
	"net/http/httputil"
	"net/url"
	"os"
	"strings"
	"sync/atomic"
	"time"
)

type Route struct {
	Prefix    string
	Upstreams []*url.URL
	Filters   []Filter
}

type Gateway struct {
	routes []*routeHandler
}

type routeHandler struct {
	prefix  string
	handler http.Handler
}

func New(routes []Route) *Gateway {
	g := &Gateway{}
	for _, r := range routes {
		g.routes = append(g.routes, &routeHandler{prefix: r.Prefix, handler: chain(newProxy(r), r.Filters...)})
	}
	return g
}

func (g *Gateway) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	var best *routeHandler
	for _, rh := range g.routes {
		if strings.HasPrefix(r.URL.Path, rh.prefix) && (best == nil || len(rh.prefix) > len(best.prefix)) {
			best = rh
		}
	}
	if best == nil {
		http.Error(w, `{"error":"no route"}`, http.StatusNotFound)
		return
	}
	best.handler.ServeHTTP(w, r)
}

func newProxy(route Route) http.Handler {
	var next atomic.Uint64
	pick := func() *url.URL { return route.Upstreams[int(next.Add(1)-1)%len(route.Upstreams)] }
	return &httputil.ReverseProxy{
		// Rewrite replaces the old Director. It sees the incoming request (In) and builds the outgoing one (Out),
		// and hop-by-hop headers are removed for us.
		Rewrite: func(pr *httputil.ProxyRequest) {
			pr.SetURL(pick())
			pr.Out.URL.Path = "/" + strings.TrimPrefix(strings.TrimPrefix(pr.In.URL.Path, route.Prefix), "/")
			pr.SetXForwarded()
		},
		Transport: &failover{
			upstreams: route.Upstreams,
			base: &http.Transport{
				MaxIdleConnsPerHost:   64,
				IdleConnTimeout:       90 * time.Second,
				DialContext:           (&net.Dialer{Timeout: 200 * time.Millisecond}).DialContext,
				ResponseHeaderTimeout: 100 * time.Millisecond,
			},
		},
		ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
			status := http.StatusBadGateway
			if errors.Is(err, os.ErrDeadlineExceeded) || strings.Contains(err.Error(), "timeout awaiting response headers") {
				status = http.StatusGatewayTimeout
			}
			http.Error(w, `{"error":"upstream unavailable"}`, status)
		},
	}
}

// failover retries an idempotent request once on another upstream when the connection itself failed.
// A POST is never retried: the first attempt may have been processed.
type failover struct {
	upstreams []*url.URL
	base      http.RoundTripper
}

func (f *failover) RoundTrip(req *http.Request) (*http.Response, error) {
	resp, err := f.base.RoundTrip(req)
	var op *net.OpError
	if err == nil || len(f.upstreams) < 2 || !errors.As(err, &op) || op.Op != "dial" {
		return resp, err
	}
	if req.Method != http.MethodGet && req.Method != http.MethodHead {
		return resp, err
	}
	for _, u := range f.upstreams {
		if u.Host == req.URL.Host {
			continue
		}
		retry := req.Clone(req.Context())
		retry.URL.Host = u.Host
		return f.base.RoundTrip(retry)
	}
	return resp, err
}

Points worth reading closely.

Matching. The gateway picks the route with the longest matching prefix, so /orders/export can have its own policy beside /orders. A table scan is fine for a few dozen routes; use the standard ServeMux patterns or a radix tree if you have hundreds.

Rewrite, not Director. ReverseProxy has two hooks for changing the outgoing request. Director is the older one, and it has a trap: because it receives the outgoing request after headers were copied, a hop-by-hop header listed in Connection can be reintroduced. Rewrite (added in Go 1.20) gets a ProxyRequest with the inbound In and outbound Out separate, and the proxy removes hop-by-hop headers and the headers it controls before calling it. SetURL points the request at the chosen upstream and SetXForwarded adds X-Forwarded-For, -Host and -Proto from the connection, discarding anything the client claimed. Use Rewrite for new code.

Prefix stripping. The route owns /orders, but the service does not know it lives there. The code trims the prefix before forwarding, so /orders/42 reaches the service as /42. Decide this per route and write it in the table, not in the service.

Timeouts. The transport has a dial timeout and a ResponseHeaderTimeout. Without them a stuck upstream holds a client connection open indefinitely. The error handler turns a header timeout into 504 and every other failure into 502, with a body that reveals nothing about the upstream address. Choose values from the route's expected latency, not one global number.

Connection reuse. MaxIdleConnsPerHost: 64 is the setting that decides whether the gateway runs out of ports under load, as the previous article's benchmark found. Do not use the default transport in a gateway.

Failover. failover wraps the transport. When the connection to the chosen upstream fails to dial, and the method is GET or HEAD, it repeats the request against another upstream once. The restriction is the point: a dial failure means the request never reached the service, so retrying is safe for any method in theory, but a gateway cannot always tell a dial failure from a failure after the upstream started working. Limiting retries to idempotent methods keeps it from ever running a payment twice. For other methods, return the error and let the client decide with an idempotency key.

Running it

The demonstration starts three upstreams (A, B and a slow one), wires two routes, and sends requests as different clients.

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"net/url"
	"strings"
	"time"
)

func upstream(name string, delay time.Duration) *httptest.Server {
	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(delay)
		fmt.Fprintf(w, `{"backend":%q,"path":%q,"user":%q,"scopes":%q,"request_id":%q,"has_api_key":%t,"forwarded_for":%q}`,
			name, r.URL.Path, r.Header.Get("X-User-Id"), r.Header.Get("X-Scopes"), r.Header.Get("X-Request-Id"),
			r.Header.Get("X-API-Key") != "", r.Header.Get("X-Forwarded-For"))
	}))
}

func must(raw string) *url.URL { u, _ := url.Parse(raw); return u }

func main() {
	a, b, slow := upstream("A", 0), upstream("B", 0), upstream("slow", 400*time.Millisecond)
	defer a.Close()
	defer slow.Close()

	keys := map[string]Client{"key-ada": {"ada", "orders:read"}, "key-bob": {"bob", "reports:read"}, "key-carol": {"carol", "orders:read"}}
	gw := httptest.NewServer(New([]Route{
		{Prefix: "/orders", Upstreams: []*url.URL{must(a.URL), must(b.URL)},
			Filters: []Filter{RequestID(), StripIdentity(), APIKey(keys), RateLimit(0.0001, 3), RequireScope("orders:read")}},
		{Prefix: "/reports", Upstreams: []*url.URL{must(slow.URL)},
			Filters: []Filter{RequestID(), StripIdentity(), APIKey(keys), RequireScope("reports:read")}},
	}))
	defer gw.Close()

	do := func(label, method, path string, headers map[string]string) {
		req, _ := http.NewRequest(method, gw.URL+path, nil)
		for k, v := range headers {
			req.Header.Set(k, v)
		}
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			fmt.Printf("%-28s error %v\n", label, err)
			return
		}
		body, _ := io.ReadAll(resp.Body)
		resp.Body.Close()
		fmt.Printf("%-28s %d %s\n", label, resp.StatusCode, strings.TrimSpace(string(body)))
	}
	ada := map[string]string{"X-API-Key": "key-ada"}
	do("no key", "GET", "/orders/42", nil)
	do("ada, spoofing X-User-Id", "GET", "/orders/42", map[string]string{"X-API-Key": "key-ada", "X-User-Id": "admin"})
	do("ada again (round robin)", "GET", "/orders/43", ada)
	do("ada third request", "GET", "/orders/44", ada)
	do("ada fourth request", "GET", "/orders/45", ada)
	do("bob lacks orders scope", "GET", "/orders/42", map[string]string{"X-API-Key": "key-bob"})
	do("no route", "GET", "/unknown", ada)

	b.Close() // one upstream dies
	time.Sleep(10 * time.Millisecond)
	carol := map[string]string{"X-API-Key": "key-carol"}
	do("upstream B is down: GET", "GET", "/orders/50", carol)
	do("upstream A: POST", "POST", "/orders/50", carol)
	do("upstream B is down: POST", "POST", "/orders/50", carol)
	do("bob: slow upstream", "GET", "/reports/1", map[string]string{"X-API-Key": "key-bob"})
}

no key                       401 {"error":"invalid api key"}
ada, spoofing X-User-Id      200 {"backend":"A","path":"/42","user":"ada","scopes":"orders:read","request_id":"req-002","has_api_key":false,"forwarded_for":"127.0.0.1"}
ada again (round robin)      200 {"backend":"B","path":"/43","user":"ada","scopes":"orders:read","request_id":"req-003","has_api_key":false,"forwarded_for":"127.0.0.1"}
ada third request            200 {"backend":"A","path":"/44","user":"ada","scopes":"orders:read","request_id":"req-004","has_api_key":false,"forwarded_for":"127.0.0.1"}
ada fourth request           429 {"error":"rate limit exceeded"}
bob lacks orders scope       403 {"error":"insufficient scope"}
no route                     404 {"error":"no route"}
upstream B is down: GET      200 {"backend":"A","path":"/50","user":"carol","scopes":"orders:read","request_id":"req-007","has_api_key":false,"forwarded_for":"127.0.0.1"}
upstream A: POST             200 {"backend":"A","path":"/50","user":"carol","scopes":"orders:read","request_id":"req-008","has_api_key":false,"forwarded_for":"127.0.0.1"}
upstream B is down: POST     502 {"error":"upstream unavailable"}
bob: slow upstream           504 {"error":"upstream unavailable"}

Reading the results

Authentication. With no key, 401. With key-ada, the request arrives at backend A with user: ada, scopes: orders:read, has_api_key: false, a request id and forwarded_for: 127.0.0.1.

Spoofing is neutralised. The second request sent X-User-Id: admin along with ada's key. The backend still saw ada. The strip filter deleted the client's header, and the key filter then set the real one. If you reverse those two in the route list, the spoof gets through; try it.

Balancing. The next two requests went to B then A, plain round robin. Counting is per route and atomic.

Rate limit. The limiter allows a burst of 3 and refills slowly. Ada's fourth request got 429.

Scope. Bob's key is valid but lacks orders:read, so 403. A 403 here is right because bob is authenticated and not allowed; a 401 would tell him to try another key.

Unknown path. 404 from the gateway itself, before any upstream is contacted.

Failover. After B was stopped, carol's GET, which the round robin sent to B, still returned 200 from A. Her next POST went to A and worked. The POST after that went to B, and the gateway returned 502 without retrying. This is the retry rule doing what it should.

Timeout. Bob's request to the slow service, which sleeps for 400 ms against a 100 ms limit, returned 504.

Passing identity downstream

The gateway tells services who the caller is by setting X-User-Id and X-Scopes. That works only while the network between them is trusted: a service reachable by anything other than the gateway would accept any header. There are three ways to tighten it, in rising order of effort.

Network policy. Only the gateway's address can reach the services (security groups, Kubernetes network policies, a private subnet). Cheap and effective, and it fails open if someone later exposes a service by mistake.

Mutual TLS between the gateway and each service, so a service accepts connections only from a client certificate that the gateway holds. The headers are then trustworthy because the channel is. mTLS configuration in Go shows the setup.

A signed assertion. The gateway signs a short-lived token (a JWT with a 30 second lifetime, say) carrying the user id, scopes and request id, and puts it in an X-Gateway-Assertion header. Services verify the signature with the gateway's public key and ignore the plain headers. This survives a service being reachable directly, because a forged request has no valid signature, and it gives each service a verifiable record of what the gateway decided. It costs a signature per request, which is microseconds with Ed25519, and a key rotation story.

Whichever you pick, document the contract between the gateway and the services in one place, and test that a service rejects a request that did not come through the gateway. An identity header that nobody checks is a vulnerability waiting for a reason.

Balancing beyond round robin

Round robin is fine when upstreams are identical and requests cost about the same. Two refinements are worth knowing before you reach for a bigger tool.

Least in-flight requests. Keep an atomic counter per upstream, increment when a request starts and decrement when it finishes, and pick the upstream with the smallest count. It adapts to slow instances without any health check, because a struggling instance accumulates in-flight requests and gets fewer new ones. The cost is a few lines in pick and a wrapper that decrements, including on errors and cancellation.

Consistent hashing by a key such as the tenant or session id keeps one client on one instance, which helps caches and stateful connections. The cost is uneven load when keys are skewed, and a rebalancing event whenever an instance joins or leaves.

Keep the choice per route in the table, defaulting to round robin. The more interesting problem is not the algorithm but the list: upstreams come and go, and a static slice does not know. If you run on Kubernetes, point the route at the service name and let kube-proxy or the mesh pick the pod, and use the gateway's list only for the services you place yourself. For the broader picture, the load balancer article goes deeper.

Configuration and reload

Hard-coded routes are fine for a demonstration and wrong for anything you operate. A small YAML or JSON file is enough:

routes:
  - prefix: /orders
    upstreams: [http://orders-1:8080, http://orders-2:8080]
    filters: [request_id, strip_identity, api_key, "rate_limit:60/min", "scope:orders:read"]

Map each filter name to a constructor in a registry and build the chain from that. Validate the file at start-up and refuse to run on an unknown filter or an unparseable upstream, because a typo should fail the deploy and not a request.

For reloading, build a complete new Gateway from the file and store it in an atomic.Pointer[Gateway]. The server's handler loads the pointer on every request and calls it. Requests in flight keep using the old value, new ones get the new one, and there is no lock on the hot path. Reload on SIGHUP or when the file changes, and keep the old configuration if the new one fails validation. Log which version of the configuration is active, and expose it on an admin port so you can tell what a running instance believes.

What this does not do yet

The honest list, in rough order of how soon you will need each item.

1. Health checks. A dead upstream keeps receiving its share of traffic, and only failover for GETs hides it. Add active checks (probe an endpoint every few seconds) or passive ones (eject an upstream after N consecutive failures), and skip unhealthy ones in pick. See health check endpoints.
2. Circuit breaking. Stop sending to a failing service for a while instead of waiting for each timeout. A breaker per upstream, as in the circuit breaker article.
3. Configuration. Routes are Go values here. Load them from a file, validate them, and reload on a signal by swapping an atomic.Pointer to the new route table so requests in flight finish on the old one.
4. Observability. Metrics per route (request count, status classes, latency histogram) and structured access logs with the request id. Propagate trace context so a trace spans the gateway and the service, as in tracing with OpenTelemetry.
5. TLS and HTTP/2. Terminate TLS in front or in the gateway process, and set server-side timeouts, ReadHeaderTimeout above all (server timeouts).
6. Streaming and upgrades. ReverseProxy handles WebSocket upgrades and flushes streaming responses when FlushInterval is set to a negative value. Test the routes that need them.
7. Request size limits with http.MaxBytesReader as a filter, and header size limits on the server.
8. Graceful shutdown, draining connections before exit, as in graceful shutdown.

Testing it

Everything above is testable with httptest, exactly as main does: start backends that echo what they received, start the gateway, make requests, assert on what the backend saw. The assertions that matter are on the backend side: that the spoofed header is gone, that the credential is gone, that the path is rewritten, that a POST is not retried. Those are the security properties, and a unit test of a filter in isolation will not catch an ordering mistake in a route. Write one test per route that asserts the whole chain.

Keep it small

The temptation with a gateway is to keep adding features until it is a bad version of Envoy. Decide up front what it will never do: no business logic, no response transformation beyond headers, no aggregation. When you need those, build a separate service. When you need retries, health checks, outlier detection and observability to the standard of a mature proxy, stop and adopt one. The standard library documentation for httputil.ReverseProxy lists every hook the code above uses, and the Go 1.20 release notes explain why Rewrite replaced Director. The Forwarded and X-Forwarded-For conventions the proxy sets are described in RFC 7239.