Memory Mapped Files in Go and the Trade Offs You Accept

Memory mapping asks the kernel to make a file look like a region of memory. The program then reads the file by indexing a slice, with no Read call and no buffer of its own, and the kernel loads pages from disk (or from its cache) when they are first touched. It is the technique behind databases like LMDB and BoltDB and behind many search indexes, and in Go it is a few lines with golang.org/x/sys/unix (the standard library does not expose mmap on its own). The question is when it pays. This article maps a 512 MB file and measures a sequential scan, two million random reads and the cost of opening, then lists what you accept in exchange for the speed, because the list is not short.

It continues reading large files quickly and bufio, where the baseline ReadAt and buffer numbers come from, and connects to syscall overhead. The cornerstone is the Go performance guide.

The lab

The program writes a 512 MB log file, opens it, maps it read only and shared with unix.Mmap, and runs three experiments, each the best of three runs. As in the other file articles, the file was just written, so it sits in the operating system's page cache, and after the first run the mapping's pages are present too. This favours mmap: a cold file would take page faults and disk reads, which the next section describes.

package main

import (
	"bufio"
	"bytes"
	"encoding/binary"
	"fmt"
	"math/rand/v2"
	"os"
	"time"

	"golang.org/x/sys/unix"
)

const size = 512 << 20

func generate(path string) {
	f, _ := os.Create(path)
	w := bufio.NewWriterSize(f, 4<<20)
	line := []byte("2000-01-01T00:00:00Z INFO request served path=/library/example status=200 bytes=18342\n")
	for n := 0; n < size; n += len(line) {
		w.Write(line)
	}
	w.Flush()
	f.Close()
}

func best(f func()) time.Duration {
	b := time.Hour
	for i := 0; i < 3; i++ {
		t := time.Now()
		f()
		if d := time.Since(t); d < b {
			b = d
		}
	}
	return b
}

var sink uint64

func main() {
	path := "big.log"
	generate(path)
	defer os.Remove(path)
	f, _ := os.Open(path)
	defer f.Close()
	st, _ := f.Stat()
	n := int(st.Size())
	data, err := unix.Mmap(int(f.Fd()), 0, n, unix.PROT_READ, unix.MAP_SHARED)
	if err != nil {
		panic(err)
	}
	defer unix.Munmap(data)
	fmt.Printf("%d MB file, in the operating system cache\n\n", n>>20)

	fmt.Println("sequential scan: count the newlines")
	d := best(func() { sink = uint64(bytes.Count(data, []byte{'\n'})) })
	fmt.Printf("  %-44s %9v %6.1f GB/s  (%d)\n", "mmap, bytes.Count on the mapping", d.Round(time.Millisecond), float64(n)/1e9/d.Seconds(), sink)
	buf := make([]byte, 256<<10)
	d = best(func() {
		var c int
		for off := int64(0); off < int64(n); {
			k, _ := f.ReadAt(buf, off)
			if k == 0 {
				break
			}
			c += bytes.Count(buf[:k], []byte{'\n'})
			off += int64(k)
		}
		sink = uint64(c)
	})
	fmt.Printf("  %-44s %9v %6.1f GB/s  (%d)\n", "ReadAt in 256 KB chunks", d.Round(time.Millisecond), float64(n)/1e9/d.Seconds(), sink)

	const reads = 2_000_000
	r := rand.New(rand.NewPCG(5, 6))
	offs := make([]int, reads)
	for i := range offs {
		offs[i] = r.IntN(n - 8)
	}
	fmt.Println()
	fmt.Printf("random access: %d reads of 8 bytes at random offsets\n", reads)
	d = best(func() {
		var s uint64
		for _, o := range offs {
			s += binary.LittleEndian.Uint64(data[o:])
		}
		sink = s
	})
	fmt.Printf("  %-44s %9v %6.0f ns/read\n", "mmap, index the slice", d.Round(time.Millisecond), float64(d.Nanoseconds())/reads)
	var b8 [8]byte
	d = best(func() {
		var s uint64
		for _, o := range offs {
			f.ReadAt(b8[:], int64(o))
			s += binary.LittleEndian.Uint64(b8[:])
		}
		sink = s
	})
	fmt.Printf("  %-44s %9v %6.0f ns/read\n", "pread through File.ReadAt (one syscall each)", d.Round(time.Millisecond), float64(d.Nanoseconds())/reads)
	d = best(func() {
		var s uint64
		for _, o := range offs {
			s += uint64(data[o])
		}
		sink = s
	})
	fmt.Printf("  %-44s %9v %6.0f ns/read\n", "mmap, one byte per access", d.Round(time.Millisecond), float64(d.Nanoseconds())/reads)

	fmt.Println()
	fmt.Println("opening cost: map the file and touch one byte, against reading one 4 KB page")
	d = best(func() {
		g, _ := os.Open(path)
		m, _ := unix.Mmap(int(g.Fd()), 0, n, unix.PROT_READ, unix.MAP_SHARED)
		sink = uint64(m[n/2])
		unix.Munmap(m)
		g.Close()
	})
	fmt.Printf("  %-44s %9v\n", "open + mmap 512 MB + touch one byte + unmap", d.Round(time.Microsecond))
	page := make([]byte, 4096)
	d = best(func() {
		g, _ := os.Open(path)
		g.ReadAt(page, int64(n/2))
		g.Close()
	})
	fmt.Printf("  %-44s %9v\n", "open + ReadAt 4 KB + close", d.Round(time.Microsecond))
}

512 MB file, in the operating system cache

sequential scan: count the newlines
  mmap, bytes.Count on the mapping                  11ms   49.9 GB/s  (6242686)
  ReadAt in 256 KB chunks                           45ms   12.0 GB/s  (6242686)

random access: 2000000 reads of 8 bytes at random offsets
  mmap, index the slice                             13ms      7 ns/read
  pread through File.ReadAt (one syscall each)     1.29s    645 ns/read
  mmap, one byte per access                          8ms      4 ns/read

opening cost: map the file and touch one byte, against reading one 4 KB page
  open + mmap 512 MB + touch one byte + unmap       60µs
  open + ReadAt 4 KB + close                        14µs

Results

Random access is where mmap wins by a mile

Reading 8 bytes at a random offset through the mapping took 7 ns: an index into a slice, a load, and at most a TLB lookup. The same read with ReadAt took 645 ns, 92 times longer, because each call is a system call into the kernel and a copy out of the page cache. For a program that does millions of small random reads, an index structure, a key value store, a lookup table on disk, that is the difference between a usable and an unusable design. The access is also free of allocation and of any buffer management.

The slice view makes the code simple. Reading a record at offset o is binary.LittleEndian.Uint64(data[o:]). There is no read loop, no handling of short reads, and the data can be handed to functions that take []byte without copying.

Sequential scans gain a little

The scan was four times faster through the mapping (49.9 GB/s against 12.0 GB/s) and the reason is the copy. ReadAt copies each 256 KB chunk from the page cache into your buffer, and the mapping lets bytes.Count read the cache pages directly. This gain is real and the least important one: a sequential read through a buffer of 32 KB to 1 MB already runs at memory speed, as the large file article showed, and on a cold file both are limited by the disk. If a sequential scan is all you do, mmap is a modest improvement for considerable risk.

Opening costs more than a small read

Mapping the whole file, touching one byte and unmapping took 60 µs, against 14 µs to read a page with ReadAt. The mapping setup is a system call plus the page table work, and the first touch is a page fault. For one off reads of small pieces of many files, ReadAt is cheaper, and mmap pays off when the mapping is kept open and used many times.

What you accept

Errors become crashes. data[o] cannot return an error. If the file is truncated while it is mapped (by another process, or by the disk dropping a network mount), touching a page that no longer has backing makes the kernel send SIGBUS, and a Go program that gets it dies with an unrecoverable fault, unless it was built to catch the signal and set debug.SetPanicOnFault(true), which turns it into a recoverable panic on that goroutine. With ReadAt the same event is an io.EOF or an error you handle. A file you own and never truncate is safe to map. A file other programs write is not.

The garbage collector does not know the memory. The mapped region is outside the Go heap, so it does not count toward the heap goal, the memory limit (GOMEMLIMIT) or heap profiles. It does count toward the process's resident size, since pages the kernel has loaded appear in the RSS that the container and its out of memory killer measure (RSS against the heap). A service that maps a 40 GB file has an RSS that rises as pages are touched, and in a container with a memory limit, mapped file pages are reclaimable but can still push the cgroup into pressure.

The first touch is slow. The numbers here are for a file in the page cache. On a cold file every first access to a page is a page fault that reads from disk, at the disk's latency, 100 µs on an SSD and milliseconds on a spinning disk and on network storage, and random access over a file larger than RAM is a stream of page faults. ReadAt pays the same disk cost, and the kernel's readahead helps sequential access in both cases. Mapping removes system call overhead; it does not remove the disk.

It is not portable in the standard library. The lab uses unix.Mmap, so it runs on Linux, macOS and the BSDs, and Windows needs a different API (CreateFileMapping). Libraries such as golang.org/x/exp/mmap (read only) and github.com/edsrzf/mmap-go wrap it. Plan for the platform.

Writes are subtle. A writable shared mapping writes through to the file when the kernel decides, and msync forces it. A crash can leave a file in a partly written state unless the format is designed for it. Read only mappings avoid this and are the common use.

You must unmap. unix.Munmap releases the region. Forgetting it leaks address space, and using the slice after Munmap is an access to unmapped memory and faults. Tie the lifetime of the slice to a type with a Close.

Large mappings and 32 bit systems. A mapping needs address space. On 64 bit systems a terabyte is fine. On a 32 bit system a 2 GB file cannot be mapped at once.

When mmap is the right choice

1. Random access into a big read only file, such as an index, a lookup table, a trained model or a database page file. This is the case where the 92 times is available.
2. Many processes reading the same file. The page cache pages are shared, so ten processes mapping one 5 GB file use 5 GB, not 50.
3. Zero copy parsing of a format that can be read in place, without building Go objects from it.
4. A working set that fits in RAM and is read repeatedly, so the faults are paid once.

When it is the wrong choice:

• A single sequential pass: use a buffer of 64 KB to 1 MB.
• A file that others may truncate or rewrite, or that lives on a network share.
• Many small files, where the setup cost per file dominates.
• Code that needs to return errors for read failures rather than crash.
• Programs that must run on Windows without extra work.

A safe pattern

type MappedFile struct {
	data []byte
}

func Open(path string) (*MappedFile, error) {
	f, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	defer f.Close() // the mapping stays valid after the descriptor closes
	st, err := f.Stat()
	if err != nil {
		return nil, err
	}
	data, err := unix.Mmap(int(f.Fd()), 0, int(st.Size()), unix.PROT_READ, unix.MAP_SHARED)
	if err != nil {
		return nil, err
	}
	return &MappedFile{data: data}, nil
}

func (m *MappedFile) Close() error { return unix.Munmap(m.data) }

An empty file cannot be mapped (a length of zero is an error), so check the size first. Keep the type's methods small and bounds checked, so a bad offset is an error you return and not a panic. Document that the file must not be modified while open.

Checklist

• Random reads: 7 ns mapped against 645 ns with ReadAt on a cached file, 92 times.
• Sequential scans: 49.9 GB/s against 12.0 GB/s, and not worth the risk by themselves.
• Mapping has a setup cost (60 µs against 14 µs), so keep the mapping open.
• A truncated file means SIGBUS, and read errors become faults.
• Mapped pages count in RSS and not in the Go heap.
• Cold files cost page faults; measure on your storage.
• Wrap it in a type with Close, and prefer read only.

The call is documented in the mmap(2) manual page and the Go wrapper in golang.org/x/sys/unix.