Goroutine Stack Dumps in Go and How to Read Them Fast

When a Go service hangs, stops serving, or slowly eats memory, the first useful artifact is a list of what every goroutine is doing right now. The runtime can print it in one move, and it is the closest thing Go has to a thread dump. The dump looks overwhelming the first time (hundreds of stacks, each a dozen lines), but it has a structure that makes it quick to read: one bracketed word per goroutine that tells you what it is waiting for.

This article shows three ways to get a dump, builds one from a program with a goroutine in each common state, and explains how to read each state, how to search a very large dump, and how to find the goroutine that matters. It belongs to the concurrency patterns cluster and is the practical companion to goroutine leaks and deadlocks. Everything ran on Go 1.25.6.

Getting a dump

SIGQUIT. Sending the signal QUIT to a running Go program makes the runtime print all goroutine stacks to standard error and exit with status 2. From a terminal, Ctrl plus the backslash key does it, and from outside, kill -QUIT <pid>. It is the zero-setup option and has the obvious cost that it kills the process, which is fine for a hung program you were going to restart anyway and a poor choice for a production service that is still partly serving. In Kubernetes the container's log then contains the dump. The GOTRACEBACK environment variable controls how much a crash prints: GOTRACEBACK=all includes every goroutine, which is the useful setting for services, and GOTRACEBACK=system adds runtime frames.

The pprof profile. pprof.Lookup("goroutine").WriteTo(w, 2) writes the full dump to any writer without stopping anything. The debug argument picks the format: 2 is the complete listing with every stack, 1 groups identical stacks and counts them, and 0 writes a binary profile for go tool pprof. With net/http/pprof imported, GET /debug/pprof/goroutine?debug=2 serves the same, which is how most services expose it (on an internal port).

runtime.Stack. runtime.Stack(buf, true) fills a byte slice with the stacks of all goroutines when the second argument is true. It is what you call from a watchdog or a test helper: "if this test hangs for 30 seconds, dump the goroutines and fail".

A dump with every common state

This program parks one goroutine in each of the commonly seen wait states, waits a moment, then prints the dump. Each helper has a one-line body, so the frame that names it shows what the goroutine is blocked on:

package main

import (
	"net"
	"os"
	"runtime/pprof"
	"sync"
	"time"
)

func waitForJob(jobs <-chan int) { <-jobs }
func sendResult(out chan<- int)  { out <- 1 }
func pickOne(a, b <-chan int) {
	select {
	case <-a:
	case <-b:
	}
}
func lockHeld(mu *sync.Mutex)      { mu.Lock() }
func napTime()                     { time.Sleep(time.Hour) }
func waitGroup(wg *sync.WaitGroup) { wg.Wait() }
func acceptLoop(ln net.Listener)   { ln.Accept() }

func main() {
	jobs, out := make(chan int), make(chan int)
	a, b := make(chan int), make(chan int)
	var mu sync.Mutex
	mu.Lock() // held forever, so lockHeld blocks
	var wg sync.WaitGroup
	wg.Add(1) // never Done, so waitGroup blocks
	ln, _ := net.Listen("tcp", "127.0.0.1:0")

	go waitForJob(jobs)
	go sendResult(out)
	go pickOne(a, b)
	go lockHeld(&mu)
	go napTime()
	go waitGroup(&wg)
	go acceptLoop(ln)
	time.Sleep(100 * time.Millisecond)

	pprof.Lookup("goroutine").WriteTo(os.Stdout, 2)
}

goroutine 1 [running]:
runtime/pprof.writeGoroutineStacks({0x100d9f908, 0x14000056020})
	runtime/pprof/pprof.go:756 +0x6c
runtime/pprof.writeGoroutine({0x100d9f908?, 0x14000056020?}, 0x80802c6a0e7c3c3e?)
	runtime/pprof/pprof.go:745 +0x2c
runtime/pprof.(*Profile).WriteTo(0x100d23d30?, {0x100d9f908?, 0x14000056020?}, 0x0?)
	runtime/pprof/pprof.go:371 +0x144
main.main()
	example.test/lab/main.go:42 +0x2d0

goroutine 8 [chan receive]:
main.waitForJob(...)
	example.test/lab/main.go:11
created by main.main in goroutine 1
	example.test/lab/main.go:33 +0x144

goroutine 9 [chan send]:
main.sendResult(...)
	example.test/lab/main.go:12
created by main.main in goroutine 1
	example.test/lab/main.go:34 +0x184

goroutine 10 [select]:
main.pickOne(...)
	example.test/lab/main.go:14
created by main.main in goroutine 1
	example.test/lab/main.go:35 +0x1cc

goroutine 11 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
	runtime/sema.go:95 +0x28
internal/sync.(*Mutex).lockSlow(0x14000010190)
	internal/sync/mutex.go:149 +0x170
internal/sync.(*Mutex).Lock(...)
	internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
	sync/mutex.go:46
main.lockHeld(...)
	example.test/lab/main.go:19
created by main.main in goroutine 1
	example.test/lab/main.go:36 +0x20c

goroutine 12 [sleep]:
time.Sleep(0x34630b8a000)
	runtime/time.go:363 +0x150
main.napTime()
	example.test/lab/main.go:20 +0x28
created by main.main in goroutine 1
	example.test/lab/main.go:37 +0x218

goroutine 13 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x0?, 0x0?)
	runtime/sema.go:114 +0x38
sync.(*WaitGroup).Wait(0x140000101a0)
	sync/waitgroup.go:206 +0xa8
main.waitGroup(...)
	example.test/lab/main.go:21
created by main.main in goroutine 1
	example.test/lab/main.go:38 +0x258

goroutine 14 [IO wait]:
internal/poll.runtime_pollWait(0x10109ea00, 0x72)
	runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(0x14000124000?, 0x0?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000124000)
	internal/poll/fd_unix.go:613 +0x21c
net.(*netFD).accept(0x14000124000)
	net/fd_unix.go:161 +0x28
net.(*TCPListener).accept(0x1400007a040)
	net/tcpsock_posix.go:159 +0x24
net.(*TCPListener).Accept(0x1400007a040)
	net/tcpsock.go:380 +0x2c
...

The first goroutine listed is the one that took the dump, in state running. Every other goroutine has a different bracket, and that bracket is the wait reason the runtime recorded when the goroutine parked. Here is the meaning of each one in the output.

Reading the states

chan receive and chan send. The goroutine is blocked on <-ch or ch <- v. The frame under it names your function and the line. For a receive, ask who is supposed to send or close that channel; for a send, who is supposed to read. A chan receive (nil chan) or chan send (nil chan) means the channel variable was never initialized, which blocks forever by design (nil channels).

select. Blocked in a select with no ready case. select (no cases) is a bare select {}, a deliberate park that you meet in main of servers. The line number in your frame tells you which select it is, and you read its cases to see what would wake it.

sync.Mutex.Lock, sync.RWMutex.RLock, sync.RWMutex.Lock. Waiting for a lock. Older Go versions printed semacquire for these; the clearer names arrived in newer releases, which is a reason to check your Go version when comparing a dump to an article. The question is who holds the lock, and the dump will not say directly. Look for the goroutine that is not blocked on this lock and whose stack contains the same type's method. A pile of goroutines on one Lock line, plus one goroutine somewhere else (in an HTTP call, a database query, a sleep) with the same mutex's method in its stack, is the textbook contended-lock-held-across-I/O case (lock contention).

sync.WaitGroup.Wait. Waiting for the counter to reach zero. If it is stuck, some Done is missing, usually because a goroutine returned early on an error path without deferring it, or Add was called after Wait started.

sync.Cond.Wait. A condition variable wait. The goroutine waits for a Signal or Broadcast.

sleep. In time.Sleep. Fine unless you did not mean to sleep that long. The duration in the bracket is how long it has been parked, not how long it will sleep.

IO wait. Waiting on network I/O through the runtime's poller: an Accept, a Read from a socket, a Write that has filled the send buffer. This is the most common healthy state in a server (idle connections), and the most common suspicious one in a hang (a client or upstream that never answers). If thousands of goroutines sit in IO wait inside your HTTP client with no deadline, you have the cause of a leak.

syscall. In a system call that did not go through the poller, such as a blocking file read or a cgo call. A handful is normal; many in a row indicates slow disks or a cgo library that blocks (blocking syscalls).

runnable and running. The goroutine is ready to execute or executing. A dump with hundreds of runnable goroutines and few cores is a CPU-saturated program, not a hang.

semacquire, chan receive (scan), GC assist wait, GC sweep wait. Runtime-internal states. A few show up in any dump; they matter only when numerous.

Durations: the signal for leaks

When a goroutine has been blocked a while, the bracket adds how long: [chan receive, 14 minutes]. The runtime only prints the duration once it passes a minute, so its presence is itself information: a goroutine blocked for minutes in a service that handles requests in milliseconds is not part of a request. Scan a large dump for minutes and you have the list of candidate leaks in seconds. Ten thousand goroutines stuck in [chan send, 3 hours] with the same created by line is the signature of the fan-out that never drains (stopping goroutines).

Searching a large dump

A busy service has thousands of goroutines. Three habits keep the dump readable.

First, use debug=1 when you want counts. It groups goroutines with identical stacks, prints how many share each, and sorts them, so "7,214 goroutines at this stack" is a single line. That is the quickest way to spot a leak by volume, and it is also smaller to ship around.

Second, filter with grep. The created by lines show where each goroutine came from, so grep -c "created by main.(*Server).handle" counts one family. Searching for minutes or hours isolates the long waiters. Searching for your package name hides the runtime frames you do not care about.

Third, read from the bottom of each stack up to understand intent and from the top down to see the wait. The top frame is where it is parked; the bottom frame, created by, is who started it. For a leak, the created by line tells you which function to open, and the parked line tells you what the goroutine was waiting for that never came.

From the dump to the fix

The dump answers "what is everyone waiting for", and the pattern in the answer usually points at the bug. All goroutines blocked, none running, and the program not exiting is a deadlock; in that case the runtime itself prints fatal error: all goroutines are asleep - deadlock! if every goroutine is parked in a way it can detect. Many goroutines in one state at one line is a leak or a bottleneck. One goroutine holding a lock while others wait on it is contention. A goroutine in IO wait on a call with no timeout is an unbounded wait waiting to happen (a timeout on every blocking call).

For continuing diagnosis, the pprof documentation describes the debug levels, the runtime package documents Stack, and the diagnostics page on go.dev places the goroutine profile among the other tools. For the next step after you find a leak, the goleak article shows how to catch the same bug in tests.