Buffered IO in Go and When bufio Actually Matters Most

bufio is the standard library's answer to a simple problem: system calls are expensive, and a program that makes one for every few bytes spends its time crossing into the kernel. A bufio.Writer collects small writes in a memory buffer and sends them in large pieces, and a bufio.Reader fetches large pieces and hands them out a byte or a line at a time. The advice to use it is everywhere, and so is the advice to stop using it when it does nothing. This article measures what it saves on writes and reads of a file, and in the cases where nobody should wrap anything, so you can tell which situation you are in.

It continues reading large files quickly, whose buffer size results apply here, and fast CSV parsing. Syscall costs themselves are in syscall overhead. The cornerstone is the Go performance guide.

Writing: the lab

Half a million log lines of 64 bytes go to a file, 32 MB in all, written seven ways. Each is the best of three runs, so the numbers show the steady state and not the first cold run. The file lives in the temporary directory, so the kernel's page cache absorbs the writes and the measurement is dominated by the cost of calling into the kernel, not by a disk.

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
	"strconv"
	"time"
)

const lines = 500_000

var line = []byte("2000-01-01T00:00:00Z INFO request served status=200 bytes=18342\n")

func timeIt(f func()) time.Duration {
	best := time.Hour
	for i := 0; i < 3; i++ {
		t := time.Now()
		f()
		if d := time.Since(t); d < best {
			best = d
		}
	}
	return best
}

func main() {
	path := "out.log"
	defer os.Remove(path)
	mb := float64(lines*len(line)) / 1e6
	fmt.Printf("write %d lines of %d bytes (%.1f MB) to a file\n", lines, len(line), mb)
	row := func(name string, f func(w io.Writer)) {
		d := timeIt(func() {
			fl, _ := os.Create(path)
			f(fl)
			fl.Close()
		})
		fmt.Printf("  %-52s %9v %8.0f MB/s %8.0f ns/line\n", name, d.Round(time.Millisecond), mb/d.Seconds(), float64(d.Nanoseconds())/lines)
	}
	row("os.File.Write per line (one syscall each)", func(w io.Writer) {
		for i := 0; i < lines; i++ {
			w.Write(line)
		}
	})
	for _, bs := range []int{4 << 10, 64 << 10, 1 << 20} {
		bs := bs
		row(fmt.Sprintf("bufio.Writer, %d byte buffer", bs), func(w io.Writer) {
			bw := bufio.NewWriterSize(w, bs)
			for i := 0; i < lines; i++ {
				bw.Write(line)
			}
			bw.Flush()
		})
	}
	row("fmt.Fprintf per line, unbuffered", func(w io.Writer) {
		for i := 0; i < lines; i++ {
			fmt.Fprintf(w, "line %d status=%d\n", i, 200)
		}
	})
	row("fmt.Fprintf per line, bufio 64 KB", func(w io.Writer) {
		bw := bufio.NewWriterSize(w, 64<<10)
		for i := 0; i < lines; i++ {
			fmt.Fprintf(bw, "line %d status=%d\n", i, 200)
		}
		bw.Flush()
	})
	row("AppendInt into one []byte, bufio 64 KB", func(w io.Writer) {
		bw := bufio.NewWriterSize(w, 64<<10)
		var b []byte
		for i := 0; i < lines; i++ {
			b = append(b[:0], "line "...)
			b = strconv.AppendInt(b, int64(i), 10)
			b = append(b, " status=200\n"...)
			bw.Write(b)
		}
		bw.Flush()
	})

	fmt.Println()
	fmt.Println("read that file one byte at a time and count the newlines")
	fl, _ := os.Create(path)
	bw := bufio.NewWriterSize(fl, 1<<20)
	for i := 0; i < 20000; i++ {
		bw.Write(line)
	}
	bw.Flush()
	fl.Close()
	small := float64(20000*len(line)) / 1e6
	rrow := func(name string, f func(r io.Reader) int) {
		var n int
		d := timeIt(func() {
			fl, _ := os.Open(path)
			n = f(fl)
			fl.Close()
		})
		fmt.Printf("  %-52s %9v %8.1f MB/s  (%d lines)\n", name, d.Round(time.Microsecond), small/d.Seconds(), n)
	}
	rrow("File.Read of 1 byte at a time", func(r io.Reader) int {
		var b [1]byte
		n := 0
		for {
			_, err := r.Read(b[:])
			if err != nil {
				return n
			}
			if b[0] == '\n' {
				n++
			}
		}
	})
	rrow("bufio.Reader.ReadByte", func(r io.Reader) int {
		br := bufio.NewReader(r)
		n := 0
		for {
			c, err := br.ReadByte()
			if err != nil {
				return n
			}
			if c == '\n' {
				n++
			}
		}
	})
	rrow("bufio.Reader.ReadByte, 64 KB buffer", func(r io.Reader) int {
		br := bufio.NewReaderSize(r, 64<<10)
		n := 0
		for {
			c, err := br.ReadByte()
			if err != nil {
				return n
			}
			if c == '\n' {
				n++
			}
		}
	})
	rrow("bufio.Reader wrapped around a bufio.Reader", func(r io.Reader) int {
		br := bufio.NewReader(bufio.NewReader(r))
		n := 0
		for {
			c, err := br.ReadByte()
			if err != nil {
				return n
			}
			if c == '\n' {
				n++
			}
		}
	})
}

write 500000 lines of 64 bytes (32.0 MB) to a file
  os.File.Write per line (one syscall each)                788ms       41 MB/s     1575 ns/line
  bufio.Writer, 4096 byte buffer                            23ms     1383 MB/s       46 ns/line
  bufio.Writer, 65536 byte buffer                           16ms     1991 MB/s       32 ns/line
  bufio.Writer, 1048576 byte buffer                          7ms     4335 MB/s       15 ns/line
  fmt.Fprintf per line, unbuffered                         836ms       38 MB/s     1671 ns/line
  fmt.Fprintf per line, bufio 64 KB                         45ms      706 MB/s       91 ns/line
  AppendInt into one []byte, bufio 64 KB                     8ms     3921 MB/s       16 ns/line

read that file one byte at a time and count the newlines
  File.Read of 1 byte at a time                        620.788ms      2.1 MB/s  (20000 lines)
  bufio.Reader.ReadByte                                  2.871ms    445.9 MB/s  (20000 lines)
  bufio.Reader.ReadByte, 64 KB buffer                    2.711ms    472.2 MB/s  (20000 lines)
  bufio.Reader wrapped around a bufio.Reader             2.867ms    446.5 MB/s  (20000 lines)

Results

One syscall per line cost 1,575 ns, and the buffer removed almost all of it. Writing each line directly meant 500,000 write calls, at about 1.5 µs each: 788 ms for 32 MB. The same loop through a 4 KB buffer made one call per 64 lines, and took 23 ms. That is 34 times faster for a change of one line of code (bufio.NewWriter(f)) and a flush at the end.

Bigger buffers help, with diminishing returns. The step from 4 KB to 64 KB took 23 ms to 16 ms, and 1 MB gave 7 ms. Most of the win arrives by 4 KB, and the rest is the number of system calls falling from 8,000 to 500 to 32. The memory cost is the buffer size per writer, so a program with thousands of writers (one per connection) should keep it near 4 to 32 KB. For one big output stream, 64 KB to 1 MB is a fine choice. The differences between these three are small compared with the 49 times between unbuffered and buffered, and a single best of three run is noisy, so do not over fit the exact sizes.

fmt.Fprintf is slow mostly because of what it writes to. Unbuffered it took 1,671 ns per line, almost exactly the syscall cost, and with a 64 KB buffer 91 ns per line. The formatting itself costs about 60 ns per call, which is the difference between the buffered Fprintf (91 ns) and the buffered write of ready bytes (32 ns). That is fine for most programs, and it is why a log line formatter that does one Fprintf per line to a raw file is a common source of "why is my logger slow".

Building the line by hand is the next 5 times. Appending the text and the integer into one reused slice with strconv.AppendInt and writing that through the buffer took 16 ns a line and 8 ms in total, 5.6 times faster than the buffered Fprintf. It is worth doing for a hot output path (a million lines a second). It is not worth doing for a report.

Reading: one byte at a time

The second half writes a 1.3 MB file and counts newlines by reading single bytes, the extreme case of small reads:

Reading a byte at a time from the file meant 1.3 million read calls, at 0.5 µs each: 621 ms for 1.3 MB, a rate that would take two minutes to read a 250 MB file. ReadByte through a bufio.Reader did the same loop in 2.9 ms, 216 times faster. This is the case for bufio in its purest form: a parser that wants to look at one byte at a time (a tokenizer, a CSV reader, a protocol decoder) gets its convenient interface from the buffer and pays the system call once per 4 KB.

The 64 KB buffer made little difference (6 percent): once the syscall count is down to 20 or 300 for the file, the loop is dominated by the call to ReadByte itself. Nesting a second bufio.Reader around the first cost nothing measurable, because bufio.NewReader returns the same reader if its argument is already a large enough *bufio.Reader. It is pointless, but harmless.

When bufio does nothing

Wrapping is not free and not always useful. Skip it when:

• The destination is already in memory. A bytes.Buffer or strings.Builder is a buffer. Writing through a bufio.Writer into one adds a copy.
• The writes are already large. A Write of 100 KB goes straight through a bufio.Writer to the underlying writer, since it is larger than the buffer. Nothing is saved.
• The library buffers for you. http.ResponseWriter buffers (4 KB), encoding/json's Encoder writes one value per call, compress/gzip writes buffered blocks, and csv.Writer is itself buffered. Do not wrap them again; call Flush where the type provides it.
• You read whole files with os.ReadFile or large chunks. A buffer between a big Read and a big consumer is a wasted copy.
• You need the data to arrive promptly. A buffered writer holds bytes until the buffer fills or you flush. Interactive protocols, log lines you want to see during a crash, and request and response exchanges over a socket need a flush after each message or no buffering at all.

The failure modes

Forgetting Flush. The last partial buffer never reaches the file, and the program exits with a truncated output and no error. Always defer w.Flush() or call it explicitly before closing the file, and check its error: the write to the underlying file happens there, so a full disk shows up as a Flush error, not as a Write error. The safe pattern is:

bw := bufio.NewWriter(f)
// ... writes ...
if err := bw.Flush(); err != nil {
	return err
}
return f.Close()

A crash loses the buffer. A panic or a kill signal discards what is in the buffer. For logs you cannot lose, flush after every line (and accept the syscall) or on a short timer, or use the kernel's buffer and accept its guarantees.

Sharing a bufio.Writer between goroutines. It is not safe for concurrent use. Guard it with a mutex, or give each goroutine its own and merge, or send lines on a channel to a single writer goroutine.

Mixing buffered and unbuffered access to the same file (writing through the bufio.Writer and directly through the file) interleaves output in the wrong order.

Choosing the size

1. Default (4 KB) is fine for most programs; 23 ms against 788 ms.
2. 32 to 64 KB for streams you produce at volume: logs, exports, network output.
3. 1 MB only for single big streams where memory is not a concern.
4. Match the medium. For a network connection, a buffer much larger than the TCP segments in flight adds latency, not speed. For a disk, buffers beyond the filesystem block size and the readahead window stop helping.

Checklist

• Unbuffered small writes cost about 1.5 µs each; a buffer cuts it to 15 to 46 ns.
• bufio.Reader.ReadByte was 216 times faster than one byte File.Read.
• Always flush, and check the flush error.
• Do not wrap a type that already buffers or that writes in large pieces.
• Do not share a bufio.Writer across goroutines.
• Measure on your medium, since a socket or disk changes the picture.

The package is documented at bufio, and the cost of a system call is explained in the Linux write(2) manual page.