Generate a Sitemap in Go for a Dynamic Website Quickly
Generate sitemaps in Go for a dynamic site: encode URLs correctly, write honest lastmod dates, split at 50,000 URLs with an index and gzip the output.
A sitemap is a letter to search engines saying: these are the pages I want you to know about, and this is when each last changed. A small static site does not need one, because crawlers find every page by following links. A dynamic site does: thousands of pages generated from a database, new ones appearing daily, some reachable only through a search form or a deep pagination chain. Without a sitemap, the pages that matter can wait weeks to be found. With a wrong one, you actively mislead the crawler, which is worse than having none.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
This article generates one in Go, with the rules that make it correct: encoding, honest dates, the size limits and the index file. It pairs with the RSS feed generator, uses pages from the Markdown blog as its example, and belongs to the web cluster that ends in a production HTTP server. The protocol is defined at sitemaps.org, and Google's sitemap overview says how its crawler treats the content. Everything ran on Go 1.25.6 with the standard library only.
What goes in, and what stays out
The format is small. A <urlset> element holds <url> entries, each with a required <loc> and optional <lastmod>. The protocol also defines <changefreq> and <priority>, and I leave both out: Google's documentation states it ignores them, and values nobody trusts are noise. Keep the file to what search engines use.
More important than the format is the list. A sitemap should contain the canonical, indexable, successful URL of each page you want in the index, and nothing else. Leave out pages that redirect, that return an error, that carry noindex, and URL variants of a page that has a canonical elsewhere (sort orders, session parameters, tracking tags). Every URL must be on the same host as the sitemap itself, with the same scheme. A sitemap that lists redirecting or non-canonical addresses wastes crawl budget and sends contradictory signals, and Search Console will report the mismatches.
The generator
package main
import (
"bytes"
"compress/gzip"
"encoding/xml"
"fmt"
"net/url"
"strings"
"time"
)
const (
maxURLs = 50000
maxBytes = 50 << 20
xmlns = "http://www.sitemaps.org/schemas/sitemap/0.9"
)
type Page struct {
Path string // "/posts/hello", already the canonical path
Modified time.Time // zero means unknown, and then no lastmod is written
}
type urlEntry struct {
Loc string `xml:"loc"`
LastMod string `xml:"lastmod,omitempty"`
}
type urlSet struct {
XMLName xml.Name `xml:"urlset"`
Xmlns string `xml:"xmlns,attr"`
URLs []urlEntry `xml:"url"`
}
type indexRef struct {
Loc string `xml:"loc"`
LastMod string `xml:"lastmod,omitempty"`
}
type sitemapIndex struct {
XMLName xml.Name `xml:"sitemapindex"`
Xmlns string `xml:"xmlns,attr"`
Sitemaps []indexRef `xml:"sitemap"`
}
// location builds the absolute, correctly encoded URL for a page, or says why it cannot.
func location(base *url.URL, path string) (string, error) {
u, err := url.Parse(path)
if err != nil {
return "", err
}
if u.Scheme != "" || u.Host != "" {
return "", fmt.Errorf("%q points at another site; a sitemap may only list URLs on its own host", path)
}
if !strings.HasPrefix(u.Path, "/") {
return "", fmt.Errorf("%q is not an absolute path", path)
}
if u.Fragment != "" {
return "", fmt.Errorf("%q has a fragment, which is never part of a page's address", path)
}
if u.RawQuery != "" {
u.RawQuery = u.Query().Encode() // percent-encodes non-ASCII and puts parameters in a stable order
}
full := base.ResolveReference(u).String()
if len(full) > 2048 {
return "", fmt.Errorf("%q is longer than 2048 characters", path[:20]+"...")
}
return full, nil
}
// Build splits pages into sitemap files of at most perFile URLs and writes an index that lists them.
func Build(baseURL string, pages []Page, perFile int) (files map[string][]byte, index []byte, skipped []error, err error) {
base, err := url.Parse(baseURL)
if err != nil {
return nil, nil, nil, err
}
perFile = min(perFile, maxURLs)
files = map[string][]byte{}
var refs []indexRef
var newest time.Time
var batch []urlEntry
var batchNewest time.Time
flush := func() error {
if len(batch) == 0 {
return nil
}
name := fmt.Sprintf("sitemap-%d.xml", len(files)+1)
var buf bytes.Buffer
buf.WriteString(xml.Header)
enc := xml.NewEncoder(&buf)
enc.Indent("", " ")
if err := enc.Encode(urlSet{Xmlns: xmlns, URLs: batch}); err != nil {
return err
}
if buf.Len() > maxBytes {
return fmt.Errorf("%s is %d bytes; the limit is %d, use a smaller perFile", name, buf.Len(), maxBytes)
}
files[name] = buf.Bytes()
ref := indexRef{Loc: base.JoinPath(name).String()}
if !batchNewest.IsZero() {
ref.LastMod = batchNewest.UTC().Format("2006-01-02")
}
refs = append(refs, ref)
batch, batchNewest = nil, time.Time{}
return nil
}
for _, p := range pages {
loc, err := location(base, p.Path)
if err != nil {
skipped = append(skipped, err)
continue
}
e := urlEntry{Loc: loc}
if !p.Modified.IsZero() {
e.LastMod = p.Modified.UTC().Format("2006-01-02")
if p.Modified.After(batchNewest) {
batchNewest = p.Modified
}
if p.Modified.After(newest) {
newest = p.Modified
}
}
batch = append(batch, e)
if len(batch) == perFile {
if err := flush(); err != nil {
return nil, nil, skipped, err
}
}
}
if err := flush(); err != nil {
return nil, nil, skipped, err
}
var buf bytes.Buffer
buf.WriteString(xml.Header)
enc := xml.NewEncoder(&buf)
enc.Indent("", " ")
if err := enc.Encode(sitemapIndex{Xmlns: xmlns, Sitemaps: refs}); err != nil {
return nil, nil, skipped, err
}
return files, buf.Bytes(), skipped, nil
}
func gzipped(b []byte) []byte {
var out bytes.Buffer
zw, _ := gzip.NewWriterLevel(&out, gzip.BestCompression)
zw.Write(b)
zw.Close()
return out.Bytes()
}Three decisions in that file are worth explaining.
location is where URLs become correct. It rejects the inputs that would produce an invalid or misleading entry: a path that is not absolute, a URL on a different host, and a fragment. It also fixes the query string. u.Query().Encode() percent-encodes non-ASCII characters (café becomes caf%C3%A9) and writes parameters in a stable order, so the same page always produces the same loc. encoding/xml then escapes the & between parameters as &, which is the XML-level escaping the protocol requires on top of URL encoding. The two are separate steps and both are needed.
lastmod is optional, and the code writes it only when Modified is non-zero. If you do not know when a page last changed, leave the element out. Writing the current time for every page on every generation, which many generators do, tells the crawler that your whole site changes every day, and once it notices the dates are not reliable it stops trusting them for the pages where they are. Use the date the content actually changed, from your data, not the time the sitemap was built. Date-only values (2026-02-03) are enough; a full timestamp is allowed when you have it.
Build splits the pages into files and writes an index. The real limit is 50,000 URLs or 50 MB uncompressed per file, whichever comes first, and a sitemap index may list up to 50,000 files, so the format scales to billions of URLs. The code checks the byte limit as well as the count, because long URLs hit 50 MB before 50,000 entries. The demo uses a limit of three URLs per file so the split is visible.
Running it
package main
import (
"fmt"
"sort"
"time"
)
func main() {
d := func(m time.Month, day int) time.Time { return time.Date(2026, m, day, 15, 4, 5, 0, time.UTC) }
pages := []Page{
{"/", d(3, 1)},
{"/posts/hello", d(1, 10)},
{"/posts/embedding-files", d(2, 3)},
{"/search?q=café&page=2", time.Time{}}, // unknown modification time: no lastmod
{"/tags/go & web", d(2, 20)},
{"/posts/relative-path-bug", d(2, 1)},
{"relative/path", d(1, 1)},
{"/posts/with-fragment#top", d(1, 1)},
{"https://other.example/x", d(1, 1)},
}
files, index, skipped, err := Build("https://notes.example", pages, 3)
if err != nil {
panic(err)
}
fmt.Println("skipped:")
for _, s := range skipped {
fmt.Println(" ", s)
}
fmt.Println("\nindex:")
fmt.Println(string(index))
names := make([]string, 0, len(files))
for n := range files {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
fmt.Printf("%s\n%s\n", n, files[n])
}
// How much does gzip save on a realistic size?
var big []Page
for i := 0; i < 5000; i++ {
big = append(big, Page{Path: fmt.Sprintf("/library/some-article-slug-number-%d", i), Modified: d(time.Month(i%12+1), i%27+1)})
}
bigFiles, _, _, _ := Build("https://notes.example", big, 50000)
raw := bigFiles["sitemap-1.xml"]
fmt.Printf("5,000 URLs: %d bytes as XML, %d bytes gzipped (%.0f%% smaller)\n", len(raw), len(gzipped(raw)), 100-100*float64(len(gzipped(raw)))/float64(len(raw)))
}skipped:
"relative/path" is not an absolute path
"/posts/with-fragment#top" has a fragment, which is never part of a page's address
"https://other.example/x" points at another site; a sitemap may only list URLs on its own host
index:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://notes.example/sitemap-1.xml</loc>
<lastmod>2026-03-01</lastmod>
</sitemap>
<sitemap>
<loc>https://notes.example/sitemap-2.xml</loc>
<lastmod>2026-02-20</lastmod>
</sitemap>
</sitemapindex>
sitemap-1.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://notes.example/</loc>
<lastmod>2026-03-01</lastmod>
</url>
<url>
<loc>https://notes.example/posts/hello</loc>
<lastmod>2026-01-10</lastmod>
</url>
<url>
<loc>https://notes.example/posts/embedding-files</loc>
<lastmod>2026-02-03</lastmod>
</url>
</urlset>
sitemap-2.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://notes.example/search?page=2&q=caf%C3%A9</loc>
</url>
<url>
<loc>https://notes.example/tags/go%20&%20web</loc>
<lastmod>2026-02-20</lastmod>
</url>
<url>
<loc>https://notes.example/posts/relative-path-bug</loc>
<lastmod>2026-02-01</lastmod>
</url>
</urlset>
5,000 URLs: 598999 bytes as XML, 20020 bytes gzipped (97% smaller)The generator skipped three of the nine inputs and said why: a relative path, a URL with a fragment, and a link to another site. Skipping and reporting is the right behavior for a batch job; a single bad row in a database should not stop the sitemap from being published, and it should not be silently included either. The returned list of errors goes to your logs, where a rising count is an alarm.
The remaining six URLs went into two files of three, listed by the index, whose lastmod for each file is the newest page inside it (the first file has 2026-03-01 because it contains the home page). The search URL with a non-ASCII query came out as ?page=2&q=caf%C3%A9, normalised and escaped, and its missing date produced no lastmod element at all. The tag go & web is percent-encoded in the path as go%20&%20web: the spaces are URL-encoded, and the ampersand is XML-escaped.
The last line shows why to compress. Five thousand typical URLs made 599 KB of XML and 20 KB gzipped, a saving of 97 percent, because every entry repeats the same tags and the same host name. Search engines accept gzipped sitemaps, with a name ending in .xml.gz, and a smaller file costs less to serve and to fetch.
From the database to the response
For a site of real size, do not hold every page in memory. Query the database in a stream (a cursor or keyset pagination, see pagination) and write each entry as you read it with the xml.Encoder, flushing a file when it reaches the limit. The structure above converts to that shape by moving the loop around rows.Next(). Generate sitemaps on a schedule or when content changes, store the files, and serve them as static assets. The alternative, building the XML on each request, makes the most crawled URL of your site the most expensive one.
Serve the files with Content-Type: application/xml, and give them a short cache lifetime and a Last-Modified, as in cache headers. Tell crawlers where the sitemap lives with a line in robots.txt:
Sitemap: https://notes.example/sitemap.xmlThe robots.txt generator and the XML sitemap generator on this site produce both files by hand for small cases. Then submit the index in Google Search Console and watch the coverage report: it tells you which listed URLs were indexed, which were excluded and why, and that report is the real test of whether the sitemap is any good.
Things the basic format does not cover
For international sites, each <url> can carry xhtml:link alternates with hreflang values, listing every language version of the page; the namespace is declared on <urlset>, and the same rule applies that alternates must point at each other. Image, video and news sitemaps use further namespaces and elements, and Google documents each. For a plain content site, none of these is needed, and the simple sitemap plus a correct set of canonical tags does the job. What matters is the discipline of the list. A sitemap is a statement you make about your site every day: that these URLs exist, are canonical and changed when you say. Keep the statement true, and the file will be useful for as long as the site is. The encoding/xml rules used here are in the package documentation.
