Blocking Syscalls and the Go Scheduler Explained Simply

Concurrency in Go depends on the runtime knowing when a goroutine cannot make progress, so that the processor can be given to another one. For channels, locks and timers the runtime always knows. For system calls it often does not, because a system call is the program asking the kernel for something, and the kernel decides when it returns. A read on a socket with no data, a read on a hung network file system, a wait on a child process: these block in the kernel, and the thread blocks with them.

The runtime has a design for this, and it is why most Go programs perform well under I/O. This article explains the design, shows two ways of waiting on the same kind of event with very different costs in threads, and lists where the design does not help. It continues from goroutines and OS threads and cgo calls, and belongs to the concurrency patterns cluster. Everything ran on Go 1.25.6.

What the runtime does

Every system call made through the syscall package goes through a pair of runtime hooks, entersyscall and exitsyscall. On entry the goroutine records that it is in a syscall and releases its claim on the processor in a lazy way: the P stays attached to the thread for a short while, in the hope that the call returns quickly. A background thread, sysmon, watches for Ps stuck in syscalls. If a P has been in one for more than about 20 microseconds (and there is other work to run), sysmon takes the P away and gives it to another thread, creating one if none is idle. The blocked thread keeps waiting in the kernel. When its syscall returns, the goroutine tries to get a P back, and if none is free it parks on the run queue and the thread goes to sleep.

The effect is that a blocked syscall costs the program one thread but not one core of Go execution: the other goroutines continue on the P that was handed off. That is also why the number of threads in a Go program can legitimately exceed GOMAXPROCS.

The netpoller: waiting without a thread

For most I/O the runtime avoids the whole mechanism. File descriptors for sockets, and pipes opened through the standard library, are put in non-blocking mode and registered with the kernel's event interface (epoll on Linux, kqueue on macOS and BSD, IOCP on Windows). A read that would block does not enter a blocking syscall; it parks the goroutine and waits for the event loop to report that the descriptor is ready. A thousand idle connections cost a thousand goroutines and no threads. This is the netpoller, and it is what the IO wait state in a goroutine dump means.

The following program compares the two ways to wait on a pipe. First fifty goroutines read from pipes created with syscall.Pipe, whose descriptors are in blocking mode, so every syscall.Read blocks a thread. Then fifty goroutines read from pipes created with os.Pipe, which are registered with the poller:

package main

import (
	"fmt"
	"os"
	"runtime/pprof"
	"sync"
	"syscall"
	"time"
)

func threads() int { return pprof.Lookup("threadcreate").Count() }

func main() {
	fmt.Println("threads at start:", threads())

	// 1. Blocking descriptors: each Read holds an OS thread.
	var wg sync.WaitGroup
	var writers []int
	for range 50 {
		var p [2]int
		if err := syscall.Pipe(p[:]); err != nil {
			panic(err)
		}
		writers = append(writers, p[1])
		wg.Go(func() {
			buf := make([]byte, 1)
			syscall.Read(p[0], buf)
		})
	}
	time.Sleep(300 * time.Millisecond)
	fmt.Println("threads with 50 blocking syscall.Read calls:", threads())
	for _, w := range writers {
		syscall.Write(w, []byte("x"))
	}
	wg.Wait()

	// 2. Pollable descriptors: waiting costs a parked goroutine, not a thread.
	before := threads()
	var files []*os.File
	for range 50 {
		r, w, err := os.Pipe()
		if err != nil {
			panic(err)
		}
		files = append(files, w)
		wg.Go(func() {
			buf := make([]byte, 1)
			r.Read(buf)
		})
	}
	time.Sleep(300 * time.Millisecond)
	fmt.Println("extra threads with 50 os.File reads:", threads()-before)
	for _, w := range files {
		w.Write([]byte("x"))
	}
	wg.Wait()
}

threads at start: 5
threads with 50 blocking syscall.Read calls: 53
extra threads with 50 os.File reads: 0

The first block of fifty goroutines raised the thread count by about fifty: each is stuck in the kernel, sysmon hands each P off, and each hand-off may need a fresh thread. The second block of fifty is the same wait, expressed through os.File, and added no threads at all, because the reads parked in the runtime instead of the kernel. Same pipes, same goroutines, a different API. The point is not that you should avoid syscall; it is that the standard library's os and net types choose the cheap path for you, and the raw syscall package does not.

Where the poller does not help

Not every descriptor can be polled. The kernel's event interfaces do not report readiness for regular files, because a regular file is always "ready" as far as epoll is concerned, even when reading it means waiting for a slow disk or a network file system. So os.ReadFile, f.Read on a regular file and f.Write to one are real blocking syscalls. Usually that is invisible, since a local SSD returns in microseconds and the P handoff never triggers. It matters when the storage is slow or stuck:

• Network file systems. A hung NFS or SMB mount can block read for minutes. Every goroutine that touches it holds a thread, and the thread count grows with the number of stuck goroutines until the process hits debug.SetMaxThreads (default 10,000) and dies with thread exhaustion.
• Slow or failing disks. The same applies on a degraded local disk or an overloaded cloud volume with exhausted IOPS.
• FUSE and other userspace file systems, where a read is a round trip to another process.
• Some terminal and device reads, and system calls such as wait4, flock and fsync, which block in the kernel without a pollable descriptor.

The symptom is a service that gets slow, then its thread count climbs in the metrics (go_threads, if you export it, or ls /proc/<pid>/task | wc -l on Linux), then it dies. A goroutine dump during the slowness shows many goroutines in state syscall, often for minutes, with your file-handling function in the stack.

What to do about it

Bound the concurrency of anything that may block in the kernel for a long time. A buffered channel as a semaphore (bounded parallelism) caps the number of threads the work can tie up, exactly as in the cgo example. The right limit is what the underlying resource can serve, not how many requests arrive.

Use deadlines where the API allows. Network connections take SetDeadline; contexts can wrap work in goroutines that you abandon. For regular files there is no deadline on a blocked read, so the practical tool is to run the call in a goroutine and stop waiting on it, accepting that the goroutine and its thread stay stuck until the kernel returns. That is a leak by another name, which is why the cap matters more than the timeout. Since Go 1.10, os.File has SetDeadline for descriptors that support polling (pipes, FIFOs, some devices), but regular files are not among them.

Keep slow storage off the request path. Read through a local cache, read in a background goroutine with a bounded queue, or move the work to a worker with a limited pool.

Prefer the os and net APIs over raw syscall for anything that can wait. They set the descriptor up for the poller. If you must call syscall directly on a blocking descriptor, remember what it costs.

Watch the thread count. A threads gauge is cheap and catches this class of problem before the limit does. runtime/metrics exposes the runtime's view, and the /sched/ metrics include latency distributions that grow when Ps are starved.

Is this a problem in practice?

For most services, no. Network I/O dominates and the poller makes it cheap. The scenario that bites is local-file-heavy work on storage that is not fast and reliable: log shippers reading from network mounts, build tools, backup agents, batch jobs walking huge trees. For those, the design above is the thing to understand. The runtime scheduler design is in proc.go, whose header comment describes the P handoff in detail, the os package documentation states which files support deadlines, and the diagnostics page at go.dev/doc/diagnostics covers the tools for seeing threads and goroutines. The related case of C code holding a thread is in cgo calls and blocked threads.