Run HTTP Calls in Parallel in Go With Sensible Limits

Fetching a list of URLs is the job that makes people discover goroutines. A hundred calls that each wait 200 ms for a response take 20 seconds in a loop and, if the target allows it, a couple of seconds in parallel. The naive version, go fetch(url) for each URL, works until the list has ten thousand entries, and then it opens ten thousand sockets, is rate limited or banned by the target, and runs out of file descriptors. This article builds the version I use: bounded, ordered, cancellable and honest about failures.

It uses errgroup from golang.org/x/sync, builds on the errgroup article and bounded concurrency with SetLimit, and sits in the concurrency patterns cluster. It applies the rules from timeouts on every blocking call. Everything ran on Go 1.25.6 with x/sync v0.19.0, against a local test server so the numbers are repeatable.

The setup

The test server answers each request after 50 ms and records the highest number of requests it ever handled at once, which is how we confirm the limit is real:

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"sync/atomic"
	"time"
)

type counting struct {
	*httptest.Server
	inflight, peak atomic.Int32
}

func newServer() *counting {
	c := &counting{}
	c.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		n := c.inflight.Add(1)
		for {
			p := c.peak.Load()
			if n <= p || c.peak.CompareAndSwap(p, n) {
				break
			}
		}
		defer c.inflight.Add(-1)
		time.Sleep(50 * time.Millisecond)
		if r.URL.Path == "/bad" {
			http.Error(w, "boom", http.StatusInternalServerError)
			return
		}
		fmt.Fprintf(w, "hello from %s", r.URL.Path)
	}))
	return c
}

The pattern

The function takes URLs and returns one result per URL, in the same order. It limits concurrency with SetLimit, shares one client, and gives every request the group's context:

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"

	"golang.org/x/sync/errgroup"
)

type Result struct {
	URL  string
	Body string
	Err  error
}

const maxBody = 1 << 20 // read at most 1 MiB of any response

func fetchOne(ctx context.Context, client *http.Client, url string) (string, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
		return "", err
	}
	resp, err := client.Do(req)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close() // always: unclosed bodies leak connections
	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("%s: %s", url, resp.Status)
	}
	body, err := io.ReadAll(io.LimitReader(resp.Body, maxBody))
	return string(body), err
}

// FetchAll fetches every URL with at most limit requests in flight. A failed URL is recorded
// in its Result and does not stop the others.
func FetchAll(ctx context.Context, client *http.Client, urls []string, limit int) []Result {
	results := make([]Result, len(urls)) // one slot per URL: no lock needed
	var g errgroup.Group
	g.SetLimit(limit)
	for i, u := range urls {
		g.Go(func() error {
			body, err := fetchOne(ctx, client, u)
			results[i] = Result{URL: u, Body: body, Err: err}
			return nil // errors live in the Result, so the group never cancels
		})
	}
	g.Wait()
	return results
}

Several choices here are deliberate. results[i] is written by exactly one goroutine, so the slice needs no mutex and the output order matches the input (an index per goroutine). g.Go returns nothing from the closure because per-URL errors are data: a crawl that loses one page out of a thousand still wants the other 999. SetLimit makes g.Go block when the limit is reached, which also slows the loop that submits work, so memory for pending closures never builds up. The body is read through io.LimitReader, because a server that returns a gigabyte, accidentally or on purpose, should not be able to exhaust your memory. And defer resp.Body.Close() is not optional: an unclosed body holds the connection, which defeats keep-alive and eventually leaks sockets (closing resources).

The comparison

The timing run fetches 20 URLs three ways: one at a time, with the limit set to 5, and with the limit set to 20. The server reports the peak concurrency each time:

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"
)

func main() {
	srv := newServer()
	defer srv.Close()
	client := &http.Client{Timeout: 5 * time.Second}

	urls := make([]string, 20)
	for i := range urls {
		urls[i] = fmt.Sprintf("%s/item/%d", srv.URL, i)
	}

	for _, limit := range []int{1, 5, 20} {
		srv.peak.Store(0)
		start := time.Now()
		results := FetchAll(context.Background(), client, urls, limit)
		failed := 0
		for _, r := range results {
			if r.Err != nil {
				failed++
			}
		}
		fmt.Printf("limit %2d: %4d ms, server saw at most %2d at once, %d failed, first body %q\n",
			limit, time.Since(start).Milliseconds()/10*10, srv.peak.Load(), failed, results[0].Body)
	}
}

limit  1: 1050 ms, server saw at most  1 at once, 0 failed, first body "hello from /item/0"
limit  5:  210 ms, server saw at most  5 at once, 0 failed, first body "hello from /item/0"
limit 20:   50 ms, server saw at most 20 at once, 0 failed, first body "hello from /item/0"

The shape of the result is the point. One at a time the batch took 1,050 ms, twenty times the per-request latency. With a limit of five it took 210 ms, four rounds of five, and the server's peak is exactly five. With a limit of twenty everything runs at once and finishes in about 50 ms, which is the best case and also the setting that would hammer a real server with twenty simultaneous connections. Returns diminish quickly in practice. Latency is rarely the only constraint; the target's capacity, its rate limits, your own file descriptors and the CPU cost of handling responses all cap the useful limit well before "all of them".

Choosing the limit

There is no universal number, but there are reliable inputs. For a third-party API, read its documentation: many state a concurrency or rate limit, and exceeding it turns into 429 responses that cost more than they save. For your own service, ask the owner what it can take from one client, and remember that other callers share it. For a crawler, per-host politeness (one or two connections per host) usually matters more than the global limit. For the machine, remember that each in-flight request is a goroutine, a socket and a TLS session; a thousand is fine on a server, ten thousand needs thought, and the default http.Transport caps idle connections per host at two (MaxIdleConnsPerHost), so a limit far above that means most connections are closed and reopened. Raise MaxIdleConnsPerHost on a custom transport to match your concurrency if you call the same host a lot.

A starting point is ten and measure. Throughput that stops improving when you raise the limit means the bottleneck moved somewhere else.

Fail fast instead

The version above collects errors. Sometimes any failure invalidates the whole result, for example when the pages together form one document, and then the right move is to stop everything on the first error. errgroup.WithContext returns a derived context that is cancelled when any function returns an error, so the in-flight requests abort and the pending ones never start:

func FetchAllOrFail(ctx context.Context, client *http.Client, urls []string, limit int) ([]string, error) {
	g, ctx := errgroup.WithContext(ctx)
	g.SetLimit(limit)
	bodies := make([]string, len(urls))
	for i, u := range urls {
		g.Go(func() error {
			body, err := fetchOne(ctx, client, u) // ctx is cancelled when a sibling fails
			bodies[i] = body
			return err
		})
	}
	if err := g.Wait(); err != nil {
		return nil, err // the first error; the others were cancelled
	}
	return bodies, nil
}

Wait returns the first non-nil error. The requests cancelled because of it fail with context canceled, which errgroup does not report since it keeps only the first. Reading bodies after Wait is safe because Wait returning orders all the writes before the read (happens-before). Choose between the two versions by asking whether a partial answer is useful. A crawler, a health check across services and a bulk metadata lookup want per-item errors. A transaction assembled from several fetches wants fail-fast.

Retries, rate limits and politeness

Real targets fail transiently, so a careful fetcher retries idempotent requests with backoff (retries with backoff), honours Retry-After on 429 and 503 responses, and spreads load with a rate limiter in addition to the concurrency limit (rate limiting). Concurrency limits how many are open at once; a rate limit caps how many start per second. They are different controls and a gentle crawler uses both. Set a User-Agent that identifies you, and for anything public, check robots.txt.

Mistakes to avoid

An unbounded go loop. It works for 20 URLs and falls over at 20,000. Sharing the loop variable on old Go: before Go 1.22 the closure captured u and i by reference and every goroutine fetched the last URL (loop variables). On a module with go 1.22 or later, as here, each iteration has its own. Creating a client per request: the connection pool is per transport, so you lose keep-alive. Forgetting the timeout: the default client has none (the timeout pattern). Reading bodies you do not need: if you only want the status, drain a small amount and close, or use HEAD.

The errgroup API is documented at pkg.go.dev/golang.org/x/sync/errgroup, and the client and transport knobs at pkg.go.dev/net/http.