Defer Inside a Loop and Why Resources Pile Up Until Exit

A defer inside a loop runs when the function ends, not when the iteration ends, so files stay open and locks stay held. See the failures and three fixes.

7 min read

Summarize in AI

Quick Summary

defer runs when the surrounding function returns, not when a loop iteration ends, so defer f.Close() inside a loop keeps every file open until the function exits and defer mu.Unlock() deadlocks on the second iteration. Move the loop body into its own function, or call Close and Unlock explicitly at the end of each iteration.

Key takeaways

Deferred calls belong to the function, so a defer in a loop body accumulates one pending call per iteration and runs them all at once when the function returns.
With a file limit of 64, a loop that deferred Close opened 61 files and then failed with too many open files; the same work with a per-iteration function opened 200.
defer mu.Unlock() inside a loop that locks each pass deadlocks on iteration two with fatal error: all goroutines are asleep.
The cleanest fix is to extract the body into a function, or a function literal called in place, so each iteration gets its own defer.
A defer in a loop is also slower than a per-iteration function in the benchmark below, about 15 ns each against about 2 ns, which rarely matters next to the leak.

The rule for defer is one line and the consequence is a bug that only appears under load: a deferred call runs when the function returns, and a loop iteration is not a function. Put defer f.Close() in a loop that processes a thousand files and you have a thousand open files at the same moment, closed together at the very end, if the program gets that far. It works on a test directory with three files and fails in production. This article reproduces the failure with real numbers, shows the second way it bites (locks), and gives the three fixes in the order I prefer them. It continues from defer and the order things run in and belongs to the Go basics path.

Tested with Go 1.25.6, macOS on Apple Silicon (arm64, Apple M1 Max); the file limit demo uses syscall and runs on Unix systems only

The leak, reproduced

Go raises the process's open-file limit to the hard limit at startup, so on a development machine you will not hit it by accident. To make the failure visible, the program below lowers the limit to 64 itself, creates 200 small files, and reads them two ways. Both functions look correct. The first defers the close in the loop. The second wraps each iteration in a function literal:

main.goGo

package main

import (
	"fmt"
	"os"
	"path/filepath"
	"strings"
	"syscall"
)

func openAllLeaky(paths []string) (int, error) {
	opened := 0
	for _, p := range paths {
		f, err := os.Open(p)
		if err != nil {
			return opened, err
		}
		defer f.Close()
		opened++
	}
	return opened, nil
}

func openAllFixed(paths []string) (int, error) {
	opened := 0
	for _, p := range paths {
		err := func() error {
			f, err := os.Open(p)
			if err != nil {
				return err
			}
			defer f.Close()
			opened++
			return nil
		}()
		if err != nil {
			return opened, err
		}
	}
	return opened, nil
}

func main() {
	limit := &syscall.Rlimit{Cur: 64, Max: 64}
	if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, limit); err != nil {
		fmt.Println(err)
		return
	}

	dir, _ := os.MkdirTemp("", "defer")
	defer os.RemoveAll(dir)

	var paths []string
	for i := range 200 {
		p := filepath.Join(dir, fmt.Sprintf("f%04d", i))
		os.WriteFile(p, []byte("x"), 0o600)
		paths = append(paths, p)
	}

	n, err := openAllLeaky(paths)
	msg := "<nil>"
	if err != nil {
		msg = err.Error()[strings.LastIndex(err.Error(), ": ")+2:]
	}
	fmt.Println("leaky:", n, "files opened, then:", msg)

	n, err = openAllFixed(paths)
	fmt.Println("fixed:", n, err)
}

Text

leaky: 61 files opened, then: too many open files
fixed: 200 <nil>

The leaky version opened 61 files and then the 62nd os.Open failed with too many open files. The three descriptors the process already held (standard input, output and error) and a few the runtime keeps explain why it is not exactly 64. The fixed version opened all 200, because each file was closed at the end of its own iteration, so there was never more than one open. The count in your run depends on the limit and on what else is open, so the number will differ on another machine. The shape will not.

Production limits are higher, commonly 1,024 or 65,536, which makes the failure arrive later and under load, as a burst of errors from a service that had been fine for days. Anything that holds a descriptor has the same problem: network connections, database rows, HTTP response bodies. A handler that loops over URLs and does resp, _ := http.Get(u); defer resp.Body.Close() keeps every connection open until the handler returns.

The second failure: locks

Resources that are not file descriptors suffer in a different way. A mutex that is locked and then deferred-unlocked in a loop never gets released between iterations:

main.goGo

package main

import (
	"fmt"
	"sync"
)

func main() {
	var mu sync.Mutex
	for i := 0; i < 2; i++ {
		mu.Lock()
		defer mu.Unlock()
		fmt.Println("iteration", i)
	}
}

Text

iteration 0
fatal error: all goroutines are asleep - deadlock!

...

The first iteration prints and schedules an unlock for the end of main. The second iteration calls Lock on a mutex that is still held by the same goroutine, and the runtime stops the program with fatal error: all goroutines are asleep - deadlock!. In a real service the runtime does not detect it, because other goroutines are alive, and the symptom is a handler that hangs forever. Mutexes in Go covers lock hygiene, and the rule that follows from this example is simple: a lock and its unlock belong in the same function and the same loop iteration.

Three fixes

Extract the body into a function. The best fix, because it gives the loop body a name and a defer scope in one move:

for _, p := range paths {
	if err := process(p); err != nil {
		return err
	}
}

func process(path string) error {
	f, err := os.Open(path)
	if err != nil {
		return err
	}
	defer f.Close()
	// ... use f
	return nil
}

This reads well, and the helper is easy to test on its own. It is what I do in nearly every case. The cost is a function call per iteration, which the compiler may inline anyway.

A function literal called in place, as the fixed version did. Use this when the body needs access to many local variables and a named helper would need a long parameter list. The literal is invoked with () right after its closing brace. It is a little more awkward to read, and it keeps the variables where they were declared.

Close explicitly at the end of the iteration. No defer at all:

for _, p := range paths {
	f, err := os.Open(p)
	if err != nil {
		return err
	}
	err = use(f)
	f.Close()
	if err != nil {
		return err
	}
}

It works and it has the weakness that every early exit in the body must remember to close, which is the reason defer exists. If the body has a continue or several returns, the explicit form accumulates cleanup calls and one of them is eventually missed. I use it only when the body is two lines long.

What it costs besides the leak

A deferred call in a loop cannot use the cheap path the compiler gives ordinary defers, since the number of calls is not known when the function is compiled. I measured 100 deferred calls in one loop against 100 iterations that each ran a function with its own defer:

main_test.goGo

package main

import "testing"

var n int

func loopDefer(k int) {
	for i := 0; i < k; i++ {
		defer func() { n++ }()
	}
}

func loopFunc(k int) {
	for i := 0; i < k; i++ {
		func() {
			defer func() { n++ }()
		}()
	}
}

func BenchmarkLoopDefer(b *testing.B) {
	for b.Loop() {
		loopDefer(100)
	}
}

func BenchmarkLoopFunc(b *testing.B) {
	for b.Loop() {
		loopFunc(100)
	}
}

Text

goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
BenchmarkLoopDefer-10    	  801835	      1492 ns/op	       0 B/op	       0 allocs/op
BenchmarkLoopDefer-10    	  808586	      1468 ns/op	       0 B/op	       0 allocs/op
BenchmarkLoopDefer-10    	  823708	      1464 ns/op	       0 B/op	       0 allocs/op
BenchmarkLoopFunc-10     	 5522767	       219.1 ns/op	       0 B/op	       0 allocs/op
BenchmarkLoopFunc-10     	 5525954	       217.1 ns/op	       0 B/op	       0 allocs/op
BenchmarkLoopFunc-10     	 5536502	       217.6 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	example.test/lab	7.466s

The loop with one big defer stack took about 1,480 ns per 100 defers, roughly 15 ns each. The per-iteration function version took about 220 ns, roughly 2 ns each. So the safe form is also the faster one here, by around seven times, and neither allocated in this benchmark. In real code the I/O inside the loop dwarfs both figures. The performance difference is a footnote, and the correctness difference is the reason to care.

When a defer in a loop is intended

Sometimes you really do want everything to stay open until the end, and defer is a convenient way to say it. A function that opens a handful of files at its start, in a loop over a fixed small list, and uses them together, is one example. A test that creates several temporary resources in a loop and cleans them all up at the end, usually with t.Cleanup, is another. Both are fine when the count is small and bounded, and a comment helps the next reader see it was deliberate. The test is whether the number of pending deferred calls depends on input size. If it does, restructure. And if you find yourself deferring cleanup inside a for that runs for the life of the program, such as an accept loop in main, you have a leak that no limit will announce until it is too late.

The specification on defer statements says that deferred functions are invoked immediately before the surrounding function returns, which is the sentence this entire article is about. The Go blog's Defer, Panic, and Recover explains the function-scope rule from first principles.

Also inPerformance