Parse Big CSV Files in Go Fast and With Little Memory

A CSV file is a good test of how a Go program spends time and memory, because the work is simple and large: split lines, split fields, convert a few of them. The standard encoding/csv package handles the full format (quoted fields, embedded commas and newlines, escaped quotes) and allocates for every record it returns. For a few thousand rows that is irrelevant. For tens of millions, it is the difference between a job that runs in seconds and one that runs for minutes while the collector works. This article parses a 1,000,000 row, 36.7 MB file six ways, to sum one column, and reports time, memory and allocations for each, so you can pick the lightest approach that is still correct for your data.

It builds on reading large files quickly, buffered IO and reducing allocations, and for splitting a file across cores see parallel CSV processing. The cornerstone is the Go performance guide.

The file and the task

A generated file with five columns (id,name,city,amount,flag) and a million data rows, written by the program itself and deleted afterwards. The task is to sum the amount column, which means reading every row, finding the fourth field and converting it to a number. The answer, 499,995,000.00, is printed for each method, and all six agree. Each method runs once, after a garbage collection, in the order shown, against a file that was just written and sits in the operating system's cache, so the measurement is about parsing and not the disk. One run per method means differences under about 10 percent are not meaningful.

package main

import (
	"bufio"
	"bytes"
	"encoding/csv"
	"fmt"
	"io"
	"os"
	"runtime"
	"strconv"
	"strings"
	"time"
)

const rows = 1_000_000

func generate(path string) {
	f, _ := os.Create(path)
	w := bufio.NewWriterSize(f, 1<<20)
	w.WriteString("id,name,city,amount,flag\n")
	for i := 0; i < rows; i++ {
		fmt.Fprintf(w, "%d,user %d,city%d,%d.%02d,%t\n", i, i%5000, i%300, i%1000, i%100, i%2 == 0)
	}
	w.Flush()
	f.Close()
}

type result struct {
	name  string
	sum   float64
	dur   time.Duration
	alloc uint64
	mallo uint64
}

func measure(name string, f func() float64) result {
	runtime.GC()
	var a, b runtime.MemStats
	runtime.ReadMemStats(&a)
	t := time.Now()
	sum := f()
	d := time.Since(t)
	runtime.ReadMemStats(&b)
	return result{name, sum, d, b.TotalAlloc - a.TotalAlloc, b.Mallocs - a.Mallocs}
}

func main() {
	path := "data.csv"
	generate(path)
	defer os.Remove(path)
	st, _ := os.Stat(path)
	fmt.Printf("%d rows, %.1f MB; sum the amount column (4th field) in each way\n\n", rows, float64(st.Size())/1e6)

	var rs []result
	rs = append(rs, measure("encoding/csv, Read per record", func() float64 {
		f, _ := os.Open(path)
		defer f.Close()
		r := csv.NewReader(f)
		r.Read()
		var sum float64
		for {
			rec, err := r.Read()
			if err == io.EOF {
				break
			}
			v, _ := strconv.ParseFloat(rec[3], 64)
			sum += v
		}
		return sum
	}))
	rs = append(rs, measure("encoding/csv, ReuseRecord=true", func() float64 {
		f, _ := os.Open(path)
		defer f.Close()
		r := csv.NewReader(f)
		r.ReuseRecord = true
		r.Read()
		var sum float64
		for {
			rec, err := r.Read()
			if err == io.EOF {
				break
			}
			v, _ := strconv.ParseFloat(rec[3], 64)
			sum += v
		}
		return sum
	}))
	rs = append(rs, measure("encoding/csv, ReuseRecord, 1 MB bufio", func() float64 {
		f, _ := os.Open(path)
		defer f.Close()
		r := csv.NewReader(bufio.NewReaderSize(f, 1<<20))
		r.ReuseRecord = true
		r.Read()
		var sum float64
		for {
			rec, err := r.Read()
			if err == io.EOF {
				break
			}
			v, _ := strconv.ParseFloat(rec[3], 64)
			sum += v
		}
		return sum
	}))
	rs = append(rs, measure("bufio.Scanner + strings.Split", func() float64 {
		f, _ := os.Open(path)
		defer f.Close()
		sc := bufio.NewScanner(f)
		sc.Scan()
		var sum float64
		for sc.Scan() {
			p := strings.Split(sc.Text(), ",")
			v, _ := strconv.ParseFloat(p[3], 64)
			sum += v
		}
		return sum
	}))
	rs = append(rs, measure("bufio.Scanner + IndexByte, no copy", func() float64 {
		f, _ := os.Open(path)
		defer f.Close()
		sc := bufio.NewScanner(f)
		sc.Buffer(make([]byte, 1<<20), 1<<20)
		sc.Scan()
		var sum float64
		for sc.Scan() {
			line := sc.Bytes()
			for i := 0; i < 3; i++ {
				line = line[bytes.IndexByte(line, ',')+1:]
			}
			end := bytes.IndexByte(line, ',')
			v, _ := strconv.ParseFloat(string(line[:end]), 64)
			sum += v
		}
		return sum
	}))
	rs = append(rs, measure("read whole file, split on newlines", func() float64 {
		b, _ := os.ReadFile(path)
		var sum float64
		first := true
		for len(b) > 0 {
			i := bytes.IndexByte(b, '\n')
			if i < 0 {
				i = len(b)
			}
			line := b[:i]
			b = b[min(i+1, len(b)):]
			if first {
				first = false
				continue
			}
			for k := 0; k < 3; k++ {
				line = line[bytes.IndexByte(line, ',')+1:]
			}
			end := bytes.IndexByte(line, ',')
			v, _ := strconv.ParseFloat(string(line[:end]), 64)
			sum += v
		}
		return sum
	}))

	fmt.Printf("%-40s %9s %10s %12s %12s  %s\n", "method", "time", "MB/s", "allocated", "allocations", "sum")
	for _, r := range rs {
		fmt.Printf("%-40s %9v %10.0f %9.1f MB %12d  %.2f\n", r.name, r.dur.Round(time.Millisecond), float64(st.Size())/1e6/r.dur.Seconds(), float64(r.alloc)/1e6, r.mallo, r.sum)
	}
}

1000000 rows, 36.7 MB; sum the amount column (4th field) in each way

method                                        time       MB/s    allocated  allocations  sum
encoding/csv, Read per record                147ms        249     115.5 MB      2000024  499995000.00
encoding/csv, ReuseRecord=true               125ms        294      35.5 MB      1000019  499995000.00
encoding/csv, ReuseRecord, 1 MB bufio        121ms        303      36.5 MB      1000025  499995000.00
bufio.Scanner + strings.Split                119ms        309     127.9 MB      2000005  499995000.00
bufio.Scanner + IndexByte, no copy            71ms        517       1.0 MB            4  499995000.00
read whole file, split on newlines            64ms        570      36.7 MB            5  499995000.00

Results

encoding/csv: correct, and the allocations are a setting

The default Read allocated 115.5 MB and two million objects for a 36.7 MB file: a []string for the record and one string for the whole line per row, and the fields point into it. It still ran at 249 MB/s, which is respectable for a parser that does everything. The first change costs nothing: r.ReuseRecord = true makes Read reuse the same slice for every row, cutting allocations in half and memory to 35.5 MB, and time by 15 percent. The caveat is real. The slice is overwritten by the next call, so code that appends records to a list, sends them on a channel, or keeps a field beyond the iteration must copy what it needs (strings.Clone for a field). The string data is still a fresh allocation per row, which is why a million allocations remain.

Giving the reader a 1 MB bufio.Reader instead of the default 4 KB buffer changed nothing measurable here (121 ms against 125 ms): the csv reader already wraps its input in a buffer, and the file read was not the bottleneck.

Scanner and Split: not better, and less correct

Splitting each line with strings.Split allocated 127.9 MB (a string for the line, a slice of five pieces) and took 119 ms, about the same as encoding/csv with reuse. It also does not handle a quoted comma or newline, so it is the worst of both. It is the version many people write first, and it is not an optimisation.

The scan that copies nothing

Using Scanner.Bytes(), which returns a slice into the scanner's buffer, and bytes.IndexByte to skip three commas and find the fourth field, took 71 ms, 517 MB/s, with four allocations in total and 1.0 MB (the scanner's buffer, which the code enlarges to 1 MB so lines up to that size work). The parsed field goes through string(line[:end]) into ParseFloat, and the compiler avoids a copy for a short lived conversion of that kind. This is twice as fast as encoding/csv and uses 3 percent of its memory, because it does the minimum: find the one field wanted and convert it.

Reading the whole file is the fastest and the most dangerous

Loading the file with os.ReadFile and slicing lines with IndexByte took 64 ms, the best time, with 5 allocations and 36.7 MB, the size of the file. It is fine for files that fit comfortably in memory and a trap for files that do not: a 40 GB export crashes the process. Its speed advantage over the scanner here is 10 percent, within the noise of a single run, so streaming gives up almost nothing.

When the fast paths are wrong

The two no-copy methods split on the comma byte. That is only correct if no field contains a comma, a quote, or a newline. A CSV exported from a spreadsheet or a database can have "Smith, John", a quoted field spanning two lines or "say ""hi""", and a byte scanner will cut those fields in the wrong place and silently sum the wrong column. If you do use the fast path:

1. Check the file once. Scan for a double quote byte (bytes.IndexByte(data, '"') is the fastest search in the standard library, 63 GB/s in fast string search). If there is none, no field is quoted and the split is safe. If there is one, use encoding/csv.
2. Validate the column count of each line and fail loudly on a mismatch.
3. Test against encoding/csv on a sample, as the lab does by comparing the sums.
4. Know the line ending. Scanner strips Windows and Unix line endings alike; a custom splitter must handle both.

Choosing

The last row is the next step beyond this lab. A 36.7 MB file split into ten parts at newline boundaries can be parsed on ten cores, and since parsing here is CPU bound and each part is independent, the speedup is close to the core count until memory bandwidth dominates. The complications are real (finding the boundary, quoted newlines, ordering of results) and need a benchmark of their own.

Cheaper wins around the parser

• Convert only what you need. strconv.ParseFloat and Atoi cost tens of nanoseconds; a million rows with five conversions each add up.
• Avoid fmt.Sscanf for fields; it is far slower than strconv.
• Process rows as they arrive and write results out incrementally. Collecting a million records in a slice is where the memory goes.
• Profile. A CPU and allocation profile of the parser will show whether csv.Reader.readRecord, the allocator or your conversion code is the cost (allocation profiles).

Checklist

• encoding/csv took 147 ms and 115.5 MB; ReuseRecord = true took 125 ms and 35.5 MB.
• Scanner with strings.Split is not an optimisation: 128 MB and no faster.
• A no copy byte scan was 71 ms with 4 allocations and is only correct for unquoted data.
• Reading the whole file saves little and risks memory.
• Check for quotes before using a split, and validate column counts.
• Measure your own file; one run per method is a rough guide.

The package is documented in encoding/csv and the scanner in bufio, and the CSV format is specified in RFC 4180.