The Cost of Syscalls in Go and How to Batch Them Away

Every Write to a file or a socket, every Read, every os.Stat and every time.Sleep that parks a thread eventually becomes a system call: a request from your program to the operating system kernel, which runs in a more privileged mode and has to be entered and left. The crossing costs far more than an ordinary call, and a program that makes millions of small ones spends its time there, not in your code. This article measures the crossing on macOS and on Linux, shows that its price is the same whatever the size of the data, and shows what batching does to it. The point is a rule you can use: count the system calls on a hot path, and make fewer, larger ones.

It follows buffered IO, where the effect shows as a 49 times difference in write time, and applies to reading large files and memory mapping, which removes the call for reads. To find out how many calls a program makes, use the tools at the end. The cornerstone is the Go performance guide.

What a system call costs

A function call in Go is a jump and a return, a couple of nanoseconds. A system call has to switch the processor from user mode to kernel mode, save and restore registers, run the kernel's code for the operation (validating arguments, finding the file, locking, doing the work), and switch back. Modern kernels add mitigations for processor vulnerabilities that make the switch more expensive. The Go runtime adds its own cost: before a blocking call it tells the scheduler that this thread is entering a system call, so that other goroutines can use the processor, and it takes that back afterwards.

The result is a fixed price per call of a few hundred nanoseconds, independent of how many bytes the call moves. The lab measures it.

The lab

The program prints the platform and measures the cheapest calls, then writes to /dev/null in several sizes. /dev/null does no work, it discards the data, so what is measured is the cost of the crossing and the file layer, the floor under any real write. It then sends the same 64 KB as 65,536 writes of one byte, 1,024 writes of 64 bytes and one write, and through a bufio.Writer. I ran it on macOS (Apple M1 Max) and in a Linux container on the same machine.

package main

import (
	"bufio"
	"flag"
	"fmt"
	"os"
	"runtime"
	"sort"
	"syscall"
	"testing"
	"time"
)

func init() {
	testing.Init()
	flag.Set("test.benchtime", "150ms")
}

func median(f func(b *testing.B)) float64 {
	const runs = 5
	xs := make([]float64, 0, runs)
	for i := 0; i < runs; i++ {
		r := testing.Benchmark(f)
		xs = append(xs, float64(r.T.Nanoseconds())/float64(r.N))
	}
	sort.Float64s(xs)
	return xs[runs/2]
}

var sinkI int
var sinkT time.Time

func main() {
	fmt.Printf("%s/%s, Go %s\n\n", runtime.GOOS, runtime.GOARCH, runtime.Version())
	null, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0)
	if err != nil {
		panic(err)
	}
	defer null.Close()

	row := func(name string, ns float64, perByte int) {
		if perByte > 0 {
			fmt.Printf("  %-46s %9.1f ns %9.2f ns/KB\n", name, ns, ns/float64(perByte)*1024)
		} else {
			fmt.Printf("  %-46s %9.1f ns\n", name, ns)
		}
	}

	fmt.Println("the price of crossing into the kernel")
	row("a plain function call (math: add and return)", median(func(b *testing.B) {
		x := 1
		for b.Loop() {
			x = add(x, 2)
		}
		sinkI = x
	}), 0)
	row("time.Now() (served from user space)", median(func(b *testing.B) {
		for b.Loop() {
			sinkT = time.Now()
		}
	}), 0)
	row("syscall.Getpid() (the cheapest system call)", median(func(b *testing.B) {
		for b.Loop() {
			sinkI = syscall.Getpid()
		}
	}), 0)
	row("os.Getpid()", median(func(b *testing.B) {
		for b.Loop() {
			sinkI = os.Getpid()
		}
	}), 0)

	fmt.Println()
	fmt.Println("write to /dev/null: the call costs the same, the bytes cost extra")
	for _, n := range []int{1, 64, 4096, 65536} {
		buf := make([]byte, n)
		row(fmt.Sprintf("File.Write of %d bytes", n), median(func(b *testing.B) {
			for b.Loop() {
				null.Write(buf)
			}
		}), n)
	}

	fmt.Println()
	fmt.Println("the same 64 KB sent as 64 KB of one byte writes, 1024 writes of 64 bytes, and one write")
	all := make([]byte, 65536)
	row("65536 writes of 1 byte", median(func(b *testing.B) {
		for b.Loop() {
			for i := 0; i < len(all); i++ {
				null.Write(all[i : i+1])
			}
		}
	}), 0)
	row("1024 writes of 64 bytes", median(func(b *testing.B) {
		for b.Loop() {
			for i := 0; i < len(all); i += 64 {
				null.Write(all[i : i+64])
			}
		}
	}), 0)
	row("1 write of 65536 bytes", median(func(b *testing.B) {
		for b.Loop() {
			null.Write(all)
		}
	}), 0)
	row("65536 one byte writes through bufio (4 KB)", median(func(b *testing.B) {
		w := bufio.NewWriterSize(null, 4096)
		for b.Loop() {
			for i := 0; i < len(all); i++ {
				w.WriteByte(all[i])
			}
			w.Flush()
		}
	}), 0)
}

//go:noinline
func add(a, b int) int { return a + b }

darwin/arm64, Go go1.25.6

the price of crossing into the kernel
  a plain function call (math: add and return)         2.2 ns
  time.Now() (served from user space)                 34.9 ns
  syscall.Getpid() (the cheapest system call)         15.1 ns
  os.Getpid()                                         12.0 ns

write to /dev/null: the call costs the same, the bytes cost extra
  File.Write of 1 bytes                              575.4 ns 589219.42 ns/KB
  File.Write of 64 bytes                             572.1 ns   9153.37 ns/KB
  File.Write of 4096 bytes                           575.1 ns    143.77 ns/KB
  File.Write of 65536 bytes                          569.2 ns      8.89 ns/KB

the same 64 KB sent as 64 KB of one byte writes, 1024 writes of 64 bytes, and one write
  65536 writes of 1 byte                         38070791.8 ns
  1024 writes of 64 bytes                         582992.9 ns
  1 write of 65536 bytes                             572.7 ns
  65536 one byte writes through bufio (4 KB)      155893.8 ns

Results

The Linux column came from the same source file built and run in the golang:1.25 image, without any change:

docker run --rm -v "$PWD":/src:ro -w /tmp golang:1.25 bash -c \
  'mkdir w && cp /src/main.go w/ && cd w && go mod init lab && go run .'

The price is per call, not per byte

A write took 575 ns on macOS and 241 ns on Linux whether it carried 1 byte or 65,536. This is the central fact. A 64 KB write costs the same as a 1 byte write, so a program that sends 64 KB in one call is 65,536 times cheaper than one that sends it a byte at a time: 573 ns against 38.07 ms on macOS, and 242 ns against 15.86 ms on Linux. The in between case, 1,024 writes of 64 bytes, took 583 µs and 247 µs, which is exactly 1,024 times the single call cost.

Real destinations add to this. A write to a file in the page cache copies the data, and the earlier bufio lab measured about 1.5 µs per direct write of 64 bytes to a file on macOS. A write to a socket runs the network stack. /dev/null is the floor: the rest is the work for the destination.

Two systems, two answers

The table shows how misleading a single number can be. syscall.Getpid() took 15.1 ns on macOS and 171.2 ns on Linux. Neither is wrong: macOS caches the process ID in its C library, so the call never enters the kernel, and the figure is that of a function call and a cache lookup. The Linux figure is a real system call, with the container's security filter in the path. A benchmark of "a system call" that uses Getpid on macOS measures nothing about the kernel. The write numbers are comparable, and there Linux was faster, 241 ns against 575 ns, because the macOS kernel does more work to reach a device file.

Treat the absolute values as properties of this machine. What carries over is the ratio: a call is one to two orders of magnitude more than a function call, and its cost does not depend on the amount of data.

time.Now() is not a system call

time.Now() took 34.9 ns on macOS and 44.0 ns on Linux. On both, the runtime reads the clock through a mechanism that does not enter the kernel (the vDSO on Linux, a shared page on macOS). That is why timestamps in a hot loop are cheap, and why a function that calls time.Now() twice per operation costs about 80 ns, not 800.

Batching, measured

If the program produces its output a byte at a time, as a formatter does, a bufio.Writer is the batching. 65,536 calls of WriteByte into a 4 KB buffer cost 155.9 µs on macOS: 16 flushes (one system call each, 9 µs in total) plus 65,536 buffer appends at about 2 ns each. That is 244 times faster than the one byte writes on macOS and 105 times on Linux, and it looks nearly the same on both systems, because most of the remaining time is user space code, not the kernel. The gain from batching is the kernel's price multiplied by the number of calls removed.

Where the calls hide

The cost shows up in code that does not look like I/O:

• Logging. A logger that calls Write on standard error for every message makes a system call per line. A buffered or asynchronous logger makes one per batch.
• Small reads from a file or socket with a 1 byte or 512 byte buffer (read buffer sizes).
• os.Stat and os.ReadDir in a loop. Each is a call. Walking a tree with a stat for every entry is the reason filepath.WalkDir (which uses directory entries that already carry the type) is faster than filepath.Walk.
• A write per response chunk in an HTTP handler. http.ResponseWriter buffers 4 KB before sending; writing and flushing after every few bytes defeats it.
• Channels and mutexes do not make system calls unless they block, and then the runtime parks the goroutine with a futex or similar. Contention that parks threads shows as system time in a profile.
• Network round trips are system calls plus the wire. A request and response per item is two to four calls per item; pipelining and batching remove them.
• fsync is a system call whose cost is the disk, in milliseconds. Calling it per record is the usual reason a write heavy program crawls.

Finding them

1. strace -c on Linux counts system calls by type and time: strace -c -f ./app prints a summary table. A write count in the millions for a small job is the signal.
2. perf trace or bpftrace for a lower overhead view on a production host.
3. dtruss or sudo fs_usage on macOS.
4. The Go CPU profile shows time in syscall.Syscall and internal/poll.(*FD).Write; a large share there with small average write sizes is the pattern (CPU profiles).
5. The execution trace shows goroutines in system calls and their duration (execution traces).
6. Count the calls yourself by wrapping the io.Writer with a counter in a test and asserting an upper bound on calls per request.

The fixes

1. Buffer writes with bufio.Writer, and flush at message boundaries.
2. Write larger pieces. Build the whole response in a buffer and send it with one call. Use net.Buffers (WriteTo) to send several slices in one writev call.
3. Read in large chunks (32 KB to 1 MB) and parse from memory.
4. Cache the answers of calls whose results do not change (a file's size, the current directory) and avoid repeat Stat calls.
5. Use batched interfaces where they exist: ReadDir over Readdirnames, sendfile through io.Copy for file to socket, epoll through the runtime's netpoller (already used for you).
6. Reduce the number of operations instead of making them cheaper: coalesce small updates, write on a timer, or aggregate before sending.

Do not go the other way and buffer everything. A buffer that holds data for seconds delays a log line you wanted on a crash, delays a response a client waits for, and loses data if the process dies. The aim is fewer calls without more latency than the use case allows.

Checklist

• A system call cost 241 to 575 ns here against 2.2 ns for a function call, whatever the data size.
• Batching 64 KB from 65,536 calls to one took 38 ms down to 573 ns on macOS.
• bufio brought byte at a time output to 156 µs.
• time.Now() is not a system call; Getpid on macOS is not either.
• Count calls with strace -c or a counting writer, and look for small average sizes.
• Flush at the right boundaries, and check flush errors.

The call mechanism is described in the syscall(2) manual page, and the Go side in the syscall package and the runtime scheduler notes.