Read the Goroutine Trace of a Production Panic in Go

Eventually a Go service panics in production, and what you have is a block of text in a log. It looks intimidating, with hex addresses and nested frames, and it contains exactly the information needed to find the bug. Reading it is a skill that takes ten minutes to learn and saves hours the first time you use it. This article takes one trace apart, then shows the variations you will meet: all goroutines, recovered and re-raised panics, and the special messages from the runtime.

It builds on recovering from panics in goroutines, reading goroutine dumps and runtime panics, and belongs to the concurrency patterns cluster. All traces here are real output from Go 1.25.6, built with -trimpath so file paths are short.

A trace, annotated

This program has a panic three calls deep inside a goroutine started by a worker, while main waits:

package main

import (
	"fmt"
	"sync"
)

type Order struct {
	Items []string
}

func total(o *Order) int {
	return len(o.Items) // o is nil: the panic happens on this line
}

func price(o *Order) int {
	return total(o) * 10
}

func handle(id int, orders map[int]*Order) {
	fmt.Println("handling", id)
	fmt.Println("price:", price(orders[id])) // orders[7] does not exist, so the value is nil
}

func main() {
	orders := map[int]*Order{1: {Items: []string{"a", "b"}}}
	var wg sync.WaitGroup
	wg.Go(func() { handle(7, orders) })
	wg.Wait()
}

handling 7
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x8 pc=0x100cb1f54]

goroutine 19 [running]:
main.total(...)
	example.test/lab/main.go:13
main.price(...)
	example.test/lab/main.go:17
main.handle(0x7, 0x140000a40f0)
	example.test/lab/main.go:22 +0x84
main.main.func1()
	example.test/lab/main.go:28 +0x24
sync.(*WaitGroup).Go.func1()
	sync/waitgroup.go:239 +0x4c
created by sync.(*WaitGroup).Go in goroutine 1
	sync/waitgroup.go:237 +0x70
exit status 2

Everything the program printed before dying is part of the story, so read from the top.

The panic line. It starts panic: runtime error: invalid memory address or nil pointer dereference, followed by a second line, [signal SIGSEGV: segmentation violation code=... addr=0x0 pc=...]. The first line is the panic value: a runtime error with a plain-language description. The signal line says the processor faulted on a tiny address, addr=0x8 here: a nil pointer plus the offset of the field being read, which is 8 bytes into Order because len(o.Items) reads the length word of the slice. A bare *p = 3 on a nil *int faults at addr=0x0, as in the second example. Either way it is a nil dereference (nil pointer dereference). For a custom panic("something") the line is just that string; for a panic with an error it prints the error's message, and a type assertion failure or index error each has a recognizable message (runtime panics).

The goroutine header. goroutine 35 [running]: identifies the goroutine that panicked. The number is an arbitrary id. running is its state at the time.

The frames, innermost first. Below the header, each function call appears as two lines: the function name with its arguments (hex values; often ... for inlined calls), and a tab-indented file:line +0xoffset. The first frames may be runtime internals; skip down to the first frame in your own code. Here that is main.total, at main.go:13, the line return len(o.Items). The frame under it is main.price (called from line 17), then main.handle, then the closure in main. That is the call chain from the crash outward: total was called by price, called by handle, called by an anonymous function in main. The bug is at the top, and the cause is usually one frame down or two: handle passed orders[id] without checking that the key exists, and the nil came from there.

The created by line. At the bottom of the stack: created by sync.(*WaitGroup).Go in goroutine 1, with a line inside sync/waitgroup.go. It tells you who started this goroutine, with a catch worth knowing about: because wg.Go starts the goroutine on your behalf, the creator frame is the library, not your call site. With a plain go func() { ... }() statement the line would be in your own file, at the go statement. When it is the library, find the call site from the other goroutines' stacks (the GOTRACEBACK=all output below shows goroutine 1 in main.main) or from the function names in the stack itself. For a goroutine that has been running for hours, this is how you find which part of the system owns it (goroutine dumps). In a trace from a pool of workers, it points at the pool's constructor, which is usually not where the bug is, but tells you which pool.

The end. exit status 2. A panic exits with status 2, the same status as a fatal runtime error; shells and orchestrators see a crash.

What the other goroutines were doing

By default the runtime prints only the goroutine that panicked. When the cause is an interaction (a goroutine blocked on a lock held by one that panicked, a producer that died while a consumer waited), you want the rest. The GOTRACEBACK environment variable controls how much is printed:

Here is the same idea with a second goroutine parked in main, run with GOTRACEBACK=all:

package main

import (
	"sync"
	"time"
)

func consume(jobs <-chan int) {
	for range jobs { // waits for work
	}
}

func produce(jobs chan<- int) {
	jobs <- 1
	time.Sleep(10 * time.Millisecond)
	var cfg map[string]string
	_ = cfg["missing"] // a read from a nil map is fine
	var p *int
	*p = 3 // the producer panics
}

func main() {
	jobs := make(chan int)
	var wg sync.WaitGroup
	wg.Go(func() { consume(jobs) })
	wg.Go(func() { produce(jobs) })
	wg.Wait()
}

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x104dd0d08]

goroutine 35 [running]:
main.produce(...)
	example.test/lab/main.go:19
main.main.func2()
	example.test/lab/main.go:26 +0x58
sync.(*WaitGroup).Go.func1()
	sync/waitgroup.go:239 +0x4c
created by sync.(*WaitGroup).Go in goroutine 1
	sync/waitgroup.go:237 +0x70

goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x14000122010?, 0x1?)
	runtime/sema.go:114 +0x38
sync.(*WaitGroup).Wait(0x14000122010)
	sync/waitgroup.go:206 +0xa8
main.main()
	example.test/lab/main.go:27 +0xd4

goroutine 34 [chan receive]:
main.consume(...)
	example.test/lab/main.go:9
main.main.func1()
	example.test/lab/main.go:25 +0x34
sync.(*WaitGroup).Go.func1()
	sync/waitgroup.go:239 +0x4c
created by sync.(*WaitGroup).Go in goroutine 1
	sync/waitgroup.go:237 +0x70
exit status 2

The trace now has several blocks separated by blank lines: the panicking goroutine (running), the consumer ([chan receive], parked in consume at the for range jobs), and main ([sync.WaitGroup.Wait], parked at wg.Wait). The extra blocks confirm the picture: the consumer was waiting for work the producer would have sent, and main was waiting for both. In a real service the block list is much longer, and the useful habit is to skim for goroutines in unexpected states: a pile of them in [sync.Mutex.Lock] points at a lock holder, and many in [IO wait] for a long time point at a slow dependency. Make GOTRACEBACK=all the default in production environments; the extra lines cost nothing until the crash, and they are the context you will want.

Recovered, then raised again

Code that recovers a panic to log it and then panics again, to let the process die, produces a distinct format. The runtime prints both panics, and marks the first as recovered:

package main

import (
	"errors"
	"fmt"
)

func risky() {
	panic(errors.New("database connection lost"))
}

func main() {
	defer func() {
		if r := recover(); r != nil {
			fmt.Println("logging before crashing:", r)
			panic(fmt.Sprintf("fatal after recover: %v", r)) // the re-panic
		}
	}()
	risky()
}

logging before crashing: database connection lost
panic: database connection lost [recovered]
	panic: fatal after recover: database connection lost

goroutine 1 [running]:
main.main.func1()
	example.test/lab/main.go:16 +0xb0
panic({0x1047a5f20?, 0x14000012060?})
	runtime/panic.go:783 +0x120
main.risky(...)
	example.test/lab/main.go:9
main.main()
	example.test/lab/main.go:19 +0x64
exit status 2

The output has panic: database connection lost [recovered], then an indented panic: fatal after recover: database connection lost, then the stack. Both messages are there, the original and the new one, so the root cause is not lost. Note a subtlety in what you see: the trace shows the stack at the second panic, the re-raise inside the deferred function, though the first panic's stack is also recorded; in the printed output the frames include main.risky because the deferred function runs on top of the panicking stack. If you re-panic with the original value (panic(r)) the output is cleaner: the same message appears twice, once marked [recovered]. When wrapping, include the original error text so a reader of the log does not need the deferred function's source.

Reading the numbers

The +0x34 after a frame is the offset of the call instruction from the start of the function. You rarely need it, since file and line already identify the statement. It matters when the build is stripped of line tables or when you have only addresses: go tool addr2line or go tool objdump with the same binary maps them back. Build with -trimpath in production to avoid leaking absolute paths on your build machine, as in this article; the module-relative path is still enough to find the file.

Arguments in the function lines (0x14000012345) are raw words and mostly useless without the binary. Go replaces unused or unavailable ones with ? after a hex value, and shows ... for inlined functions. Do not try to decode them; use the line numbers.

A short procedure for a production panic

1. Read the first line: what kind of panic is it? A nil dereference, an index out of range, a type assertion, a custom message, a closed channel?
2. Find the first frame in your own code and open that file at that line. The statement it points to is where the invalid operation happened.
3. Walk down one or two frames to find where the bad value came from. The bug is usually a caller that did not validate, not the line that crashed.
4. Read created by to learn which goroutine and which part of the system it belongs to.
5. If other goroutines are listed, skim their states for something stuck or unexpected.
6. Reproduce in a test with the inputs that reached that code, add the missing check, and keep the test.

The format of the output is documented in the runtime package overview, which describes GOTRACEBACK and the other environment variables, and the Go FAQ and the diagnostics guide place traces among the other debugging tools. For panics you decide to recover instead of letting them crash, the supervisor and the wrapper in recovering from panics in goroutines keep the stack in the error so none of this information is lost.