Build a Concurrent Web Crawler in Go From Scratch Safely

A web crawler is the classic exercise in Go concurrency because it contains every difficulty at once: work that discovers more work, a shared record of what has been seen, a limit on how hard to hit a server, and the question of how to know when you are done. The famous version is the final exercise of the Go tour. This is the version I would be willing to point at a real site, which means it is bounded, cancellable, polite and tested.

We build it in steps against a local test site, so the numbers are repeatable and nobody's server is harmed. The article builds on pipelines, worker pools, semaphores and timeouts, and sits in the concurrency patterns cluster. Everything ran on Go 1.25.6 with only the standard library.

The requirements

Write them down before writing code; most crawler bugs come from a requirement nobody stated.

1. Start from one URL and follow links to pages on the same host, never off-site.
2. Fetch each distinct page once, even if many pages link to it, and even if the site has cycles.
3. Run fetches concurrently but never more than N at a time.
4. Stop when there is nothing left to fetch, or when the context is cancelled, or when a page budget is used up.
5. Limit depth, body size and fetch time.
6. Report pages visited, with their titles and depths, and report errors without stopping.

Requirement 2 is where the shared state lives. Requirement 4 is where concurrent crawlers get stuck, because "nothing left to fetch" has to be detected while goroutines are still adding work.

Normalizing URLs

Two strings can name one page: /a#top, /a, /a? and HTTP://Host/a often do. If the visited set compares raw strings, the same page is fetched several times and a cycle through differently spelled links never ends. So every candidate goes through one function that resolves it against the page it was found on, drops the fragment, and refuses anything off-site or not HTTP:

package main

import (
	"net/url"
	"regexp"
	"strings"
)

var hrefRe = regexp.MustCompile(`(?i)<a\s[^>]*href\s*=\s*["']([^"']+)["']`)
var titleRe = regexp.MustCompile(`(?is)<title>(.*?)</title>`)

// normalize resolves ref against base and returns the canonical form of a same-host http(s)
// URL, or false if the link should not be followed.
func normalize(base *url.URL, ref string) (string, bool) {
	u, err := base.Parse(strings.TrimSpace(ref))
	if err != nil {
		return "", false
	}
	if u.Scheme != "http" && u.Scheme != "https" {
		return "", false // mailto:, javascript:, tel: and the like
	}
	if u.Host != base.Host {
		return "", false // stay on the starting site
	}
	u.Fragment = "" // /a#top and /a are one page
	return u.String(), true
}

// links returns the normalized same-host links on a page.
func links(base *url.URL, body string) []string {
	var out []string
	for _, m := range hrefRe.FindAllStringSubmatch(body, -1) {
		if u, ok := normalize(base, m[1]); ok {
			out = append(out, u)
		}
	}
	return out
}

func title(body string) string {
	if m := titleRe.FindStringSubmatch(body); m != nil {
		return strings.TrimSpace(m[1])
	}
	return ""
}

A regular expression is not an HTML parser, and a real crawler uses golang.org/x/net/html to walk the token stream, which copes with unquoted attributes, comments and <base href>. The regexp keeps this article dependency-free and is fine for the well-formed pages of our test site. The structure does not change when you swap the extractor, because everything else only needs a []string of links. Real crawlers also have to honor <base>, rel="nofollow", robots.txt and <meta name="robots">, which come up at the end.

What normalization still misses

normalize removes fragments and resolves relative links, and that is enough for the test site. Real sites present more ways to spell one page, and each one that slips through is a duplicate fetch. Trailing slashes are the common one: /docs and /docs/ may be the same page or different pages, and only the server knows. Query strings are another. ?a=1&b=2 and ?b=2&a=1 are usually the same resource, while tracking parameters such as utm_source are always noise, so crawlers sort the parameters and drop the ones they recognize. Host case and default ports (Example.com:80) need folding, which url.Parse mostly handles, and a scheme change between http and https can make one page two. Session identifiers embedded in the URL are the worst: a site that appends ?sid= to every link presents an infinite space of distinct URLs for a finite number of pages, and the page budget is what saves you. Whatever rules you adopt, put them in the one function that every candidate passes through, and test it with a table of pairs that must compare equal and pairs that must not.

Fetching one page

The fetch is the part that talks to the network, so it gets all the defensive habits at once: a context so it can be cancelled, a timeout, a body size cap, and a check on status and content type:

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"strings"
)

const maxBody = 1 << 20 // 1 MiB per page is plenty for the HTML of almost any page

func fetch(ctx context.Context, client *http.Client, pageURL string) (string, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, pageURL, nil)
	if err != nil {
		return "", err
	}
	req.Header.Set("User-Agent", "gohacks-example-crawler/1.0")
	resp, err := client.Do(req)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("%s: %s", pageURL, resp.Status)
	}
	if ct := resp.Header.Get("Content-Type"); !strings.HasPrefix(ct, "text/html") {
		return "", fmt.Errorf("%s: not HTML (%s)", pageURL, ct)
	}
	body, err := io.ReadAll(io.LimitReader(resp.Body, maxBody))
	return string(body), err
}

Identify the crawler in the User-Agent header, with a contact URL when you point it at anyone else's site. Anyone who runs a server has the right to know who is hitting it.

The crawler itself

Here is the core. Study the order of operations in visit, because the whole concurrency story is in it:

package main

import (
	"context"
	"net/http"
	"net/url"
	"sync"
	"time"
)

type Page struct {
	URL   string
	Title string
	Depth int
}

type Crawler struct {
	Client   *http.Client
	MaxDepth int
	MaxPages int
	Workers  int

	mu     sync.Mutex
	seen   map[string]bool
	pages  []Page
	errs   []error
	active int // fetches in flight right now
	peak   int // the most we ever had in flight
}

// claim marks u visited and reports whether the caller is the first to see it. The check
// and the mark are one critical section: two goroutines cannot both win.
func (c *Crawler) claim(u string) bool {
	c.mu.Lock()
	defer c.mu.Unlock()
	if c.seen[u] || (c.MaxPages > 0 && len(c.seen) >= c.MaxPages) {
		return false
	}
	c.seen[u] = true
	return true
}

// Crawl visits every reachable page up to the limits and returns what it found.
func (c *Crawler) Crawl(ctx context.Context, start string) ([]Page, []error, int) {
	c.seen = map[string]bool{}
	base, err := url.Parse(start)
	if err != nil {
		return nil, []error{err}, 0
	}
	sem := make(chan struct{}, c.Workers)
	var wg sync.WaitGroup

	var visit func(pageURL string, depth int)
	visit = func(pageURL string, depth int) {
		defer wg.Done() // one Done per Add, whatever happens below

		select {
		case sem <- struct{}{}: // take a slot, or give up if the crawl is cancelled
		case <-ctx.Done():
			return
		}
		c.mu.Lock()
		c.active++
		c.peak = max(c.peak, c.active)
		c.mu.Unlock()

		body, err := fetch(ctx, c.Client, pageURL)

		c.mu.Lock()
		c.active--
		if err != nil {
			c.errs = append(c.errs, err)
		} else {
			c.pages = append(c.pages, Page{URL: pageURL, Title: title(body), Depth: depth})
		}
		c.mu.Unlock()
		<-sem // release the slot BEFORE queuing children, so a parent never holds a slot a child needs

		if err != nil || depth >= c.MaxDepth {
			return
		}
		pageBase, _ := url.Parse(pageURL)
		for _, link := range links(pageBase, body) {
			if c.claim(link) {
				wg.Add(1) // counted before the goroutine exists, so Wait cannot return too early
				go visit(link, depth+1)
			}
		}
	}

	if c.claim(base.String()) {
		wg.Add(1)
		go visit(base.String(), 0)
	}
	wg.Wait()
	return c.pages, c.errs, c.peak
}

func newClient() *http.Client { return &http.Client{Timeout: 5 * time.Second} }

Three decisions here are the ones that make it correct.

wg.Add(1) happens before go visit, in the parent. The WaitGroup counts URLs that are queued or in flight. If the child called Add itself, there would be a window where the parent has finished (calling Done) and the count has briefly dropped to zero before the child's Add, so Wait could return while work remains. Incrementing in the parent, before it calls its own Done in the defer, means the count never reaches zero while a child exists. That is the entire termination argument, and it is why a crawler needs no "idle detection" machinery.

claim is check-and-mark in one critical section. A separate if !seen[u] { seen[u] = true } with a lock around each statement would let two goroutines both see "not seen". The page budget is enforced in the same place, by counting the entries in seen, so the crawl never queues more than MaxPages URLs.

The slot is released before the children are spawned. If a parent held its semaphore slot while starting children that need slots, then with N workers, N parents each waiting on children that cannot start would deadlock. Releasing first means the slot is held only for the duration of the fetch, which is the thing the limit is meant to bound. The goroutines for queued URLs are cheap (a few kilobytes each, parked on the semaphore), so queueing every discovered link as a goroutine is fine for tens of thousands of pages. For millions of URLs you would hold them in a frontier queue instead and start a fixed set of workers.

A site to crawl

The test site is generated in memory: 63 pages arranged as a binary tree (page i links to pages 2i+1 and 2i+2), plus a link from a leaf back to the root to make a cycle, an off-site link, a link to a page that returns 404, and a fragment link to the page itself:

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"strconv"
	"strings"
	"time"
)

func newSite() *httptest.Server {
	const pages = 63
	mux := http.NewServeMux()
	mux.HandleFunc("GET /p/{n}", func(w http.ResponseWriter, r *http.Request) {
		n, err := strconv.Atoi(r.PathValue("n"))
		if err != nil || n < 0 || n >= pages {
			http.NotFound(w, r)
			return
		}
		time.Sleep(20 * time.Millisecond) // each page takes a little while, like a real server
		var b strings.Builder
		link := func(target, text string) { fmt.Fprintf(&b, "<a href=%q>%s</a> ", target, text) }
		fmt.Fprintf(&b, "<html><head><title>Page %d</title></head><body>", n)
		for _, child := range []int{2*n + 1, 2*n + 2} {
			if child < pages {
				link(fmt.Sprintf("/p/%d", child), fmt.Sprintf("child %d", child))
			}
		}
		link(fmt.Sprintf("/p/%d#top", n), "myself") // a fragment link to itself
		if n == 62 {
			link("/p/0", "back to the start") // a cycle
			link("/p/999", "missing")         // a 404
			link("https://"+"example.org/", "elsewhere")
			link("mailto:[email protected]", "mail")
		}
		b.WriteString("</body></html>")
		w.Header().Set("Content-Type", "text/html; charset=utf-8")
		fmt.Fprint(w, b.String())
	})
	return httptest.NewServer(mux)
}

Running it

package main

import (
	"context"
	"fmt"
	"slices"
	"time"
)

func main() {
	site := newSite()
	defer site.Close()

	for _, workers := range []int{1, 2, 4, 8, 16} {
		c := &Crawler{Client: newClient(), MaxDepth: 10, Workers: workers}
		start := time.Now()
		pages, errs, peak := c.Crawl(context.Background(), site.URL+"/p/0")
		elapsed := time.Since(start)

		slices.SortFunc(pages, func(a, b Page) int { return a.Depth - b.Depth })
		fmt.Printf("workers %2d: %d pages, %d errors, peak in flight %2d, %v\n",
			workers, len(pages), len(errs), peak, elapsed.Round(10*time.Millisecond))
		if workers == 4 {
			fmt.Printf("  deepest page: %q at depth %d; error: %v\n", pages[len(pages)-1].Title, pages[len(pages)-1].Depth, errs[0])
		}
	}
}

workers  1: 63 pages, 1 errors, peak in flight  1, 1.42s
workers  2: 63 pages, 1 errors, peak in flight  2, 720ms
workers  4: 63 pages, 1 errors, peak in flight  4, 370ms
  deepest page: "Page 51" at depth 5; error: http://127.0.0.1:58146/p/999: 404 Not Found
workers  8: 63 pages, 1 errors, peak in flight  8, 230ms
workers 16: 63 pages, 1 errors, peak in flight 16, 170ms

Every configuration visited all 63 pages exactly once despite the cycle and the fragment self-links. The one error is the 404, recorded without aborting the crawl; the off-site and mailto: links were ignored; and the peak number of fetches in flight equals the configured worker count each time, which is the limit working. The timings are the interesting part. One worker took 1.42 s, which is 63 pages at 20 ms plus overhead. Two workers halved it, four quartered it (370 ms), eight gave 230 ms and sixteen gave 170 ms. The curve flattens because of the shape of the site: it is a binary tree six levels deep, so no matter how many workers you have, a page can only be fetched after its parent, and the longest chain of dependent fetches is six pages, about 120 ms plus request overhead. That is the crawler's critical path, and more workers cannot shorten it. On a real site the same effect appears as the tail of a crawl, when a few slow deep pages are all that is left.

The lesson generalizes: the useful number of workers is bounded both by what the server can take and by how wide your graph is. A crawl with sixteen workers over a site that never has more than five unfetched links at a time uses five.

Cancellation and budgets

A crawl that cannot be stopped is a liability. The ctx passed to Crawl reaches both places where a goroutine can wait: the semaphore (select on ctx.Done()) and the HTTP request (NewRequestWithContext). Cancel the context and queued visits return without fetching, in-flight fetches abort, and wg.Wait() returns promptly because every goroutine still calls Done through the defer. A deadline is the same mechanism: context.WithTimeout(ctx, time.Minute) gives a crawl that stops after a minute with whatever it found.

MaxPages and MaxDepth are the other brakes. A site with infinite URL spaces (a calendar with a "next month" link, a search page with endless filters) never runs out of new URLs, and without a budget the crawl never ends. Depth is a rough guard and a page cap is a hard one; use both.

When the goroutine per URL stops being fine

visit starts a goroutine for every discovered URL and parks it on the semaphore until a slot is free. A goroutine costs a couple of kilobytes, so a hundred thousand queued URLs cost a few hundred megabytes of stacks and the scheduler's attention, which is tolerable for a small crawl and not for a large one. The alternative design keeps the frontier as data and starts a fixed number of workers:

frontier := make(chan job, 1024) // bounded: a full frontier slows discovery, which is backpressure
for range workers {
	go func() {
		for j := range frontier {
			process(j, frontier) // may add more jobs to frontier
			pending.Done()
		}
	}()
}

The hazard is a deadlock on a full frontier: all workers are blocked sending children into the channel, and nobody is receiving. Solutions are an unbounded in-memory queue protected by a mutex and a condition variable, a spill to disk, or a per-worker local stack that sends overflow to the shared queue. The completion argument is unchanged (increment the pending counter before enqueuing) and the dedup set is the same. Past a few hundred thousand URLs the frontier is also the thing you want to persist, so that a crash does not lose the crawl, which leads to the design in a persistent job queue.

What belongs outside the crawler

The Crawler struct keeps results in memory behind a mutex. Real uses want them elsewhere: written to a database or a file as they arrive, handed to a parser, indexed. Resist adding that to visit. Give the crawler a callback or an output channel (OnPage func(Page)), and keep fetching and discovery separate from what is done with a page. The callback runs in the visiting goroutine, so it must be safe to call concurrently, or it can send to a channel owned by a single writer, the collector pattern. That boundary also keeps tests simple: the crawler tests above need no storage.

Tests

The properties worth pinning are the ones that were hard to get right: each page is fetched once, the cycle terminates, the concurrency limit holds, depth is respected, and cancellation returns quickly. These run against the same in-memory site, under -race:

package main

import (
	"context"
	"testing"
	"time"
)

func TestEachPageIsFetchedOnce(t *testing.T) {
	site := newSite()
	defer site.Close()
	c := &Crawler{Client: newClient(), MaxDepth: 10, Workers: 8}
	pages, errs, _ := c.Crawl(context.Background(), site.URL+"/p/0")
	if len(pages) != 63 {
		t.Fatalf("visited %d pages, want 63 (cycle or dedupe broken)", len(pages))
	}
	if len(errs) != 1 {
		t.Fatalf("want exactly the one 404, got %d errors: %v", len(errs), errs)
	}
	seen := map[string]bool{}
	for _, p := range pages {
		if seen[p.URL] {
			t.Fatalf("%s was visited twice", p.URL)
		}
		seen[p.URL] = true
	}
}

func TestConcurrencyLimitHolds(t *testing.T) {
	site := newSite()
	defer site.Close()
	c := &Crawler{Client: newClient(), MaxDepth: 10, Workers: 3}
	_, _, peak := c.Crawl(context.Background(), site.URL+"/p/0")
	if peak > 3 {
		t.Fatalf("peak %d exceeded the limit of 3", peak)
	}
	if peak < 3 {
		t.Fatalf("peak %d: the crawler never used its full limit", peak)
	}
}

func TestDepthLimit(t *testing.T) {
	site := newSite()
	defer site.Close()
	c := &Crawler{Client: newClient(), MaxDepth: 2, Workers: 4}
	pages, _, _ := c.Crawl(context.Background(), site.URL+"/p/0")
	if len(pages) != 7 { // 1 + 2 + 4 pages at depths 0, 1, 2
		t.Fatalf("visited %d pages with MaxDepth 2, want 7", len(pages))
	}
}

func TestCancelStopsTheCrawl(t *testing.T) {
	site := newSite()
	defer site.Close()
	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Millisecond)
	defer cancel()
	c := &Crawler{Client: newClient(), MaxDepth: 10, Workers: 2}
	start := time.Now()
	pages, _, _ := c.Crawl(ctx, site.URL+"/p/0")
	if took := time.Since(start); took > time.Second {
		t.Fatalf("crawl took %v after cancellation", took)
	}
	if len(pages) >= 63 {
		t.Fatalf("visited all %d pages despite the deadline", len(pages))
	}
}

func TestPageBudget(t *testing.T) {
	site := newSite()
	defer site.Close()
	c := &Crawler{Client: newClient(), MaxDepth: 10, Workers: 4, MaxPages: 10}
	pages, errs, _ := c.Crawl(context.Background(), site.URL+"/p/0")
	if len(pages)+len(errs) > 10 {
		t.Fatalf("fetched %d URLs with a budget of 10", len(pages)+len(errs))
	}
}

=== RUN   TestEachPageIsFetchedOnce
--- PASS: TestEachPageIsFetchedOnce (0.23s)
=== RUN   TestConcurrencyLimitHolds
--- PASS: TestConcurrencyLimitHolds (0.49s)
=== RUN   TestDepthLimit
--- PASS: TestDepthLimit (0.07s)
=== RUN   TestCancelStopsTheCrawl
--- PASS: TestCancelStopsTheCrawl (0.07s)
=== RUN   TestPageBudget
--- PASS: TestPageBudget (0.09s)
PASS
ok  	example.test/lab	2.248s

TestConcurrencyLimitHolds asserts both directions: the peak must not exceed the limit, and it must reach it, because a crawler that quietly ran one at a time would also satisfy "at most 3". The cancel test uses a short deadline and checks that the crawl returns quickly with a partial result.

Being a good citizen

The code above is polite only by accident of its size. Before it goes near a site you do not own:

• Read and obey robots.txt. Fetch it once per host, honor Disallow and Crawl-delay, and cache it. Libraries exist; do not write the parser yourself.
• Rate limit per host, in addition to the concurrency cap. A golang.org/x/time/rate limiter per host, waited on before each fetch, turns "4 at a time" into "2 requests per second" (rate limiting).
• Back off on 429 and 5xx, respecting Retry-After (retry with backoff).
• Identify yourself with a real User-Agent and a contact address.
• Reuse connections with one shared client, and always close bodies.
• Cache with conditional requests (If-None-Match, If-Modified-Since) if you revisit pages.

For further reading, the Go tour exercise on web crawlers is the original statement of the problem, the net/http package documentation covers the client behavior used here, and the robots exclusion protocol, RFC 9309 defines robots.txt precisely. The page-fetching half of the crawler is the same pattern as parallel HTTP requests.