Profile Guided Optimization in Go and Its Real Gains

Profile guided optimization (PGO) hands the compiler something it normally lacks: knowledge of which code is hot. Without a profile, the compiler decides what to inline from static rules. With one, it can spend its inlining budget on the call sites that matter and turn hot interface calls into direct calls. The feature has been stable since Go 1.21, it is nearly free to adopt, and the headline numbers in the release notes are in single digits. This article measures it on a workload that is representative of a service (JSON in, sorting, JSON out), says exactly what to expect, and records the two ways I got no gain before getting some.

It belongs with CPU profiling, whose profile it consumes, and the rest of the performance cluster. The inlining it builds on is explained in inlining in Go, and benchmarking discipline is in benchmarking properly.

What PGO does

Given a CPU profile in pprof format, the compiler changes two things:

1. More inlining at hot call sites. The normal inlining budget is small (a function costing more than 80 units is not inlined). For call sites that the profile marks as hot, the budget grows substantially, so larger functions are inlined where it pays.
2. Devirtualisation of hot interface calls. If a call through an interface almost always reaches one concrete type, the compiler emits a type check and a direct call to that type's method, with the interface call as a fallback. A direct call can then be inlined.

That is all. There is no per function tuning knob. Both transformations are conservative: they apply only where the profile shows weight, and the code remains correct for every other case. Everything is applied package by package when the package is compiled, including the standard library packages the program imports, which is why a program that spends its time in encoding/json and sort can benefit without changing a line.

Using it

The workflow is three steps:

1. Build the program normally and run it under a representative load with a CPU profile (from pprof.StartCPUProfile, or /debug/pprof/profile on a running service).
2. Save the profile as default.pgo in the directory of the main package.
3. Build with go build. Since Go 1.21 the default is -pgo=auto, which uses default.pgo if it is there. -pgo=path names a file explicitly and -pgo=off disables it.

Merge profiles from several instances or times with go tool pprof -proto a.pprof b.pprof > default.pgo. A profile from production is the best input, since it is the real distribution of work, and profiles from different versions of the code still work because the compiler matches functions by name and line offset.

The workload

The program decodes a 40 item JSON document, builds an upper cased name list and sorts it, and encodes the document again, six thousand times per round. It is the shape of a service handler. A command line argument profile makes the same binary write its own CPU profile.

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"runtime/pprof"
	"sort"
	"strings"
	"time"
)

type Item struct {
	ID    int               `json:"id"`
	Name  string            `json:"name"`
	Tags  []string          `json:"tags"`
	Attrs map[string]string `json:"attrs"`
	Price float64           `json:"price"`
}

type Doc struct {
	Title string `json:"title"`
	Items []Item `json:"items"`
}

func makeDoc() []byte {
	d := Doc{Title: "catalogue"}
	for i := 0; i < 40; i++ {
		d.Items = append(d.Items, Item{ID: i, Name: fmt.Sprintf("item-%d", i), Tags: []string{"a", "b", "c"}, Attrs: map[string]string{"k1": "v1", "k2": strings.Repeat("x", i%9)}, Price: float64(i) * 1.25})
	}
	b, _ := json.Marshal(d)
	return b
}

func work(raw []byte, n int) int {
	total := 0
	for i := 0; i < n; i++ {
		var d Doc
		if err := json.Unmarshal(raw, &d); err != nil {
			panic(err)
		}
		names := make([]string, 0, len(d.Items))
		for _, it := range d.Items {
			names = append(names, strings.ToUpper(it.Name))
		}
		sort.Strings(names)
		out, _ := json.Marshal(d)
		total += len(out) + len(names)
	}
	return total
}

func main() {
	rounds := 6
	if len(os.Args) > 1 && os.Args[1] == "profile" {
		f, _ := os.Create("default.pgo")
		pprof.StartCPUProfile(f)
		defer func() { pprof.StopCPUProfile(); f.Close() }()
		rounds = 10
	}
	raw := makeDoc()
	var sum int
	start := time.Now()
	for r := 0; r < rounds; r++ {
		sum += work(raw, 6000)
	}
	fmt.Printf("result %d, %d ms\n", sum, time.Since(start).Milliseconds())
}

set -e
go build -o app-off -pgo=off .
echo "== 1. collect a CPU profile from the real binary under a representative run"
./app-off profile > /dev/null
ls -l default.pgo | awk '{print "profile file:", $5, "bytes"}'
go build -o app-pgo -pgo=default.pgo .
med() { sort -n | awk '{a[NR]=$1} END {print a[int((NR+1)/2)]}'; }
echo "== 2. seven runs of each build, wall time of the same work"
for b in off pgo; do
  for i in 1 2 3 4 5 6 7; do ./app-$b | awk '{print $3}'; done > times-$b.txt
  echo "app-$b: median $(med < times-$b.txt) ms (min $(sort -n times-$b.txt | head -1), max $(sort -n times-$b.txt | tail -1))"
done
echo "== 3. same result, binary size"
echo "results: $(./app-off | awk '{print $2}' | tr -d ,) $(./app-pgo | awk '{print $2}' | tr -d ,)"
ls -l app-off app-pgo | awk '{print $9, $5, "bytes"}'
rm -f app-off app-pgo times-*.txt default.pgo

== 1. collect a CPU profile from the real binary under a representative run
profile file: 11542 bytes
== 2. seven runs of each build, wall time of the same work
app-off: median 2816 ms (min 2805, max 2868)
app-pgo: median 2732 ms (min 2726, max 2938)
== 3. same result, binary size
results: 135108000 135108000
app-off 3215746 bytes
app-pgo 3298514 bytes

The script builds with -pgo=off, runs the binary once under the profiler to produce default.pgo, builds again with the profile, and runs each binary seven times, printing the median, minimum and maximum of the reported wall time.

Results

Both binaries returned 135108000. By the median, PGO saved 84 ms of 2816, 3.0 percent; by the fastest runs, 79 ms, 2.8 percent. The binary grew by 82768 bytes, 2.6 percent, because more code was inlined. The profile file was 11542 bytes.

A 3 percent gain is typical and worth taking. It costs a profile file, no code changes, and a slightly larger binary; the build is a little slower. It is not a replacement for fixing the algorithm. In an earlier run of the same workload, before I fixed the benchmark's rounds, the gain was 4.7 percent, and the range between runs of one build (2805 to 2868 ms) is as large as 2 percent, so a single comparison of two runs would not have been convincing. Seven runs and the median are the minimum for a result this small, and for a real decision repeat on the target machine.

Two ways to get nothing

A profile from the wrong binary. My first version collected the profile with go test -bench -cpuprofile. The build with -pgo produced a binary of exactly the same size and speed. Compiling with -gcflags=-d=pgodebug=2 showed why: every call site had weight zero. The profile named functions like example.test/lab.run, because inside a test binary the package under test has its import path as its name, and the real binary calls it main.run. The compiler matches by symbol name, found nothing, and silently did nothing. Collect the profile from the binary you ship, or at least a binary where the package has the same name. The lab does this with a profile mode in main.

A workload where call overhead is not the cost. A second attempt called small operations through an interface in a loop, each dependent on the previous result. PGO did devirtualise the call (the debug output reported hottest callee main.add.Apply (weight 220)), and the run times were the same, 3521 against 3543 ms. The loop is latency bound: each operation waits for the result of the one before, and the processor hides the cost of a direct versus an indirect call behind the dependency chain. PGO helps when calls are a real fraction of the time, which is the case in code with many small functions, deep call chains and interfaces, the structure of most server software. It does little for a tight numeric kernel. Check with the CPU profile first: if one function is 95 percent flat and calls nothing, there is nothing for PGO to improve.

Practical notes

• Check it is applied. Binary size changing is a hint. go version -m app shows -pgo=... in the build settings. The debug flag above shows decisions per call site.
• Keep the profile fresh enough. A profile from last year's code still works but covers less of the hot paths. Refresh it when the service changes substantially, and commit it so builds are reproducible: the profile is a build input.
• CI builds must find the file. A default.pgo in the main package directory is picked up by go build and go install without flags.
• Build time. PGO builds recompile more of the dependency graph and take longer. Measure it if your CI is tight.
• Test it. The semantics are unchanged by design, and a compiler optimisation is still a compiler change. Run your tests on the PGO build too.
• Gains vary. Reports from the Go team and other users range from 2 to 14 percent. Measure on your own program.

Is it worth it

For a service where CPU is a cost line, 3 percent is a few cores per hundred, for the price of committing a file. Since the profile is something you probably collect anyway for continuous profiling, the marginal effort is small. It is not the first thing to try: allocations, algorithms and data structures and avoiding work you do not need give larger wins. It is a good last step once those are done, and it keeps paying as the code changes.

Checklist

• Collect the profile from the real main binary under a representative load.
• Save it as default.pgo in the main package directory and commit it.
• Compare builds with at least seven runs each, using the median.
• Expect a few percent, a slightly larger binary and a slower build.
• Verify that PGO was applied; a profile from a test binary does nothing.
• Look at the CPU profile first: PGO helps call heavy code, not a single hot loop.

The feature is documented in the Go PGO guide and the Go 1.21 release notes.