Read Large Files in Go Quickly Without Loading Them All

Reading a large file in Go goes wrong in two ways. Either the program loads the whole file into memory and gets killed on the first file that is bigger than the machine, or it reads in tiny pieces and spends most of its time in system calls. The fast path sits between them: a fixed buffer of a sensible size, reused for the whole file, with whatever processing you need done on each chunk. This article measures that on a 512 MB log file with ten ways of reading it, from a 512 byte buffer to ten parallel readers, and reports time, throughput and memory for each, so the choice of buffer size and method rests on numbers.

It builds on buffered IO and when bufio matters and memory mapped files, and is the foundation for fast CSV parsing. The cornerstone is the Go performance guide.

The task and the caveat

The program writes a 512 MB file of 6,242,686 identical log lines, then counts the newlines in it ten different ways, running each once. Counting lines is a stand in for any scan that has to look at every byte. The caveat matters for how to read the results: the file was just written, so it sits in the operating system's page cache, and every read is served from RAM, not from the disk. The numbers measure the cost of system calls, copies and Go code. On a cold file, a spinning disk or a network volume the disk would dominate, and several of the differences below shrink. Run the lab on your own storage before drawing conclusions about it.

package main

import (
	"bufio"
	"bytes"
	"fmt"
	"io"
	"os"
	"runtime"
	"sync"
	"time"
)

const size = 512 << 20

func generate(path string) {
	f, _ := os.Create(path)
	w := bufio.NewWriterSize(f, 4<<20)
	line := []byte("2000-01-01T00:00:00Z INFO request served path=/library/example status=200 bytes=18342\n")
	for n := 0; n < size; n += len(line) {
		w.Write(line)
	}
	w.Flush()
	f.Close()
}

type result struct {
	name  string
	lines int
	dur   time.Duration
	alloc uint64
}

func run(name string, f func() int) result {
	runtime.GC()
	var a, b runtime.MemStats
	runtime.ReadMemStats(&a)
	t := time.Now()
	n := f()
	d := time.Since(t)
	runtime.ReadMemStats(&b)
	return result{name, n, d, b.TotalAlloc - a.TotalAlloc}
}

func readBuf(path string, bufSize int) int {
	f, _ := os.Open(path)
	defer f.Close()
	buf := make([]byte, bufSize)
	lines := 0
	for {
		n, err := f.Read(buf)
		lines += bytes.Count(buf[:n], []byte{'\n'})
		if err == io.EOF {
			return lines
		}
	}
}

func main() {
	path := "big.log"
	generate(path)
	defer os.Remove(path)
	st, _ := os.Stat(path)
	fmt.Printf("count the lines of a %d MB file that is in the operating system cache\n\n", st.Size()>>20)

	var rs []result
	for _, bs := range []int{512, 4 << 10, 32 << 10, 256 << 10, 1 << 20, 8 << 20} {
		bs := bs
		rs = append(rs, run(fmt.Sprintf("Read into a %d byte buffer", bs), func() int { return readBuf(path, bs) }))
	}
	rs = append(rs, run("bufio.Scanner, default 64 KB token limit", func() int {
		f, _ := os.Open(path)
		defer f.Close()
		sc := bufio.NewScanner(f)
		n := 0
		for sc.Scan() {
			n++
		}
		return n
	}))
	rs = append(rs, run("bufio.Reader.ReadString per line", func() int {
		f, _ := os.Open(path)
		defer f.Close()
		r := bufio.NewReader(f)
		n := 0
		for {
			_, err := r.ReadString('\n')
			if err != nil {
				return n
			}
			n++
		}
	}))
	rs = append(rs, run("os.ReadFile (whole file in memory)", func() int {
		b, _ := os.ReadFile(path)
		return bytes.Count(b, []byte{'\n'})
	}))
	rs = append(rs, run("10 goroutines with ReadAt on ranges", func() int {
		const workers = 10
		chunk := st.Size() / workers
		counts := make([]int, workers)
		var wg sync.WaitGroup
		for w := 0; w < workers; w++ {
			wg.Add(1)
			go func() {
				defer wg.Done()
				f, _ := os.Open(path)
				defer f.Close()
				buf := make([]byte, 1<<20)
				off, end := int64(w)*chunk, int64(w+1)*chunk
				if w == workers-1 {
					end = st.Size()
				}
				for off < end {
					n, _ := f.ReadAt(buf[:min(int64(len(buf)), end-off)], off)
					if n == 0 {
						return
					}
					counts[w] += bytes.Count(buf[:n], []byte{'\n'})
					off += int64(n)
				}
			}()
		}
		wg.Wait()
		t := 0
		for _, c := range counts {
			t += c
		}
		return t
	}))

	fmt.Printf("%-44s %9s %9s %12s  %s\n", "method", "time", "GB/s", "allocated", "lines")
	for _, r := range rs {
		fmt.Printf("%-44s %9v %9.2f %9.1f MB  %d\n", r.name, r.dur.Round(time.Millisecond), float64(st.Size())/1e9/r.dur.Seconds(), float64(r.alloc)/1e6, r.lines)
	}
}

count the lines of a 512 MB file that is in the operating system cache

method                                            time      GB/s    allocated  lines
Read into a 512 byte buffer                      564ms      0.95       0.0 MB  6242686
Read into a 4096 byte buffer                     104ms      5.15       0.0 MB  6242686
Read into a 32768 byte buffer                     51ms     10.56       0.0 MB  6242686
Read into a 262144 byte buffer                    46ms     11.72       0.3 MB  6242686
Read into a 1048576 byte buffer                   46ms     11.79       1.0 MB  6242686
Read into a 8388608 byte buffer                   55ms      9.76       8.4 MB  6242686
bufio.Scanner, default 64 KB token limit         177ms      3.04       0.0 MB  6242686
bufio.Reader.ReadString per line                 323ms      1.66     599.3 MB  6242686
os.ReadFile (whole file in memory)               203ms      2.64     536.9 MB  6242686
10 goroutines with ReadAt on ranges               19ms     28.82      10.5 MB  6242686

Results

The buffer size

Each call to File.Read is a system call. With a 512 byte buffer the program made a million of them and took 564 ms. A 4 KB buffer, the usual default, cut that to 104 ms, 5.4 times faster. 32 KB brought it to 51 ms. Beyond that the curve is flat: 256 KB and 1 MB both took 46 ms. The rule of thumb that comes out is to use 32 KB to 1 MB, and the middle, 64 or 128 KB, is the safe choice. It is also what Go's own io.Copy uses for its internal buffer, 32 KB.

An 8 MB buffer was slower than 1 MB (55 ms against 46 ms). A buffer that does not fit in the processor's cache has to be evicted before the data in it is counted, so the cost of the bigger buffer is paid in cache misses. The buffer that wins is the largest one that stays in the cache.

Allocation per line is the real cost

bufio.Reader.ReadString with a newline delimiter returns a new string for each line. With 6.2 million lines it allocated 599.3 MB and took 323 ms, 7 times slower than the buffered read of the same file, and most of that difference is the allocator and the collector. If you need each line as a string, that is the price; if you only need to look at it, use ReadSlice or ReadLine, or a Scanner and its Bytes(), which hand you a slice of the internal buffer without allocating.

bufio.Scanner with its default settings took 177 ms and allocated nothing, because Scan reuses its buffer and Text was not called. It is the comfortable choice for line oriented work. It has one trap: a line longer than the scanner's buffer (64 KB by default) makes Scan stop with bufio.ErrTooLong, and if you ignore the error the program quietly processes part of the file. Raise the limit with scanner.Buffer(make([]byte, 0, 1<<20), 1<<20) where lines can be long, and always check scanner.Err() after the loop.

Reading it all at once

os.ReadFile took 203 ms and 536.9 MB, 4.4 times slower than the buffered read, and the memory is the size of the file. For small files it is the simplest and best option. For a file whose size you do not control it is a risk: 40 GB in memory is an out of memory kill. It is also slower here, because the runtime has to allocate and zero half a gigabyte before reading a byte.

Parallel reading

Ten goroutines, each opening the file and reading its tenth with ReadAt into a 1 MB buffer, finished in 19 ms at 28.8 GB/s, 2.4 times faster than the best single reader. Remember the caveat: this is memory bandwidth from the page cache, and the work per byte (counting a byte) is trivial. On a real disk, parallel readers often help on SSDs and NVMe (many requests in flight) and hurt on spinning disks (seeks), and on network storage depend on the latency. The complication that this lab avoids by counting is that real parsing needs ranges that start and end on line boundaries, and each worker must skip a partial first line and finish its last one.

Practical recipes

Process a file line by line, cheaply:

f, err := os.Open(path)
if err != nil {
	return err
}
defer f.Close()
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64<<10), 1<<20) // lines up to 1 MB
for sc.Scan() {
	line := sc.Bytes() // valid until the next Scan; copy it if you keep it
	handle(line)
}
return sc.Err()

Process a file in chunks, for example to hash or count:

buf := make([]byte, 128<<10)
for {
	n, err := f.Read(buf)
	process(buf[:n])
	if err == io.EOF {
		break
	}
	if err != nil {
		return err
	}
}

Process buf[:n] before looking at the error: Read can return data and an error together.

Copy or hash a file: io.Copy(dst, f) already uses a 32 KB buffer, and for file to file or file to socket copies on Linux the runtime can skip user space entirely with copy_file_range or sendfile. Do not write your own loop for a plain copy.

Choosing

1. Small file, parsed once: os.ReadFile.
2. Large file, line by line: bufio.Scanner with a raised buffer, working on Bytes().
3. Large file, raw chunks: your own buffer of 64 KB to 1 MB.
4. Need strings for each line: accept the allocation, or reduce by keeping only the fields you need.
5. Huge file, CPU bound work, fast storage: split by byte ranges across cores.
6. Random access to a huge file: consider memory mapping (mmap).

What this lab does not measure

Cold files, compressed files, network storage, files larger than RAM, and the cost of the work you do with the data. A gzip file spends most of its time in decompression, so buffer sizes barely matter. A file on a network mount is dominated by round trips, where a larger read size matters more, not less. A file that is larger than memory is read once from disk, and then the page cache stops helping. For those, read the throughput of your storage first with a tool like dd and compare it with the numbers here: a program that reads at 1 GB/s from a disk that delivers 500 MB/s cannot be made faster by any of these changes.

Checklist

• Use a read buffer of 32 KB to 1 MB: 512 bytes was 12 times slower, 8 MB was slower than 1 MB.
• Avoid a string per line unless you need it: 599 MB and 323 ms against 0 MB and 177 ms.
• Raise Scanner limits and check Err().
• Do not ReadFile a file whose size you do not control.
• Parallel readers helped from cache (19 ms); test on your storage.
• Process the bytes before checking the error from Read.

The calls are documented in the os package and bufio, and the cache effect in the Linux page cache documentation.