Set Up protoc buf and Code Generation for gRPC in Go

Getting gRPC code generation to work once is easy. Getting it to work the same way on every laptop, in CI and next year is a small amount of engineering that most projects skip until it hurts. The symptoms of skipping it are familiar: a diff of two thousand lines in generated files because someone has a newer plugin, a build that works on one machine and fails on another because protoc-gen-go is missing or the wrong version, and generated code that no longer matches the proto because someone forgot to rerun the command.

This article sets up generation so those problems cannot happen. It builds on the first gRPC service and covers four things: where the plugins come from, how import paths and output directories work, how to check in CI that generated files are current, and what the same setup looks like in Buf. It ran on Go 1.25.6 with protoc 29.3.

What protoc does and what the plugins do

protoc parses .proto files and checks them, then hands a description of the schema to plugins that write code. It does not know about Go. A flag such as --go_out=gen means "run the executable named protoc-gen-go and write its output under gen". Each plugin takes options through --<name>_opt. Two plugins cover Go gRPC: protoc-gen-go writes the message types, and protoc-gen-go-grpc writes the client and server interfaces.

The plugins are separate programs found on the PATH. That is the reproducibility problem: which version is on the PATH? The answer is to build them yourself from versions pinned in go.mod. A tools.go file with a tools build tag imports the plugin packages, which makes the module depend on them:

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

Then go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go produces the binary at exactly the pinned version, and putting bin first on the PATH for the protoc command makes protoc use it. Updating a plugin becomes a normal dependency update with a pull request. Go 1.24 added a tool directive to go.mod and go tool for running pinned tools; where your toolchain supports it, go tool protoc-gen-go is the shorter version of the same idea.

Layout and import paths

The protos live in a tree whose directory structure is their package structure:

proto/
  common/v1/money.proto
  inventory/v1/inventory.proto
gen/
  common/v1/money.pb.go
  inventory/v1/inventory.pb.go
  inventory/v1/inventory_grpc.pb.go

One proto imports another with a path relative to the include root, not to the file: import "common/v1/money.proto";. The include root is what you give to protoc -I. With -I proto, the import resolves to proto/common/v1/money.proto. Get this wrong and you see "File not found", or worse, two copies of the same file under different names, which protobuf's registry refuses at run time with a conflict message.

The two protos here:

syntax = "proto3";
package common.v1;
option go_package = "example.test/lab/gen/common/v1;commonv1";

message Money {
  string currency = 1;
  int64 units_minor = 2;
}

syntax = "proto3";
package inventory.v1;
option go_package = "example.test/lab/gen/inventory/v1;inventoryv1";

import "common/v1/money.proto";

service Inventory {
  rpc GetPrice(GetPriceRequest) returns (GetPriceResponse);
}

message GetPriceRequest { string sku = 1; }
message GetPriceResponse { common.v1.Money price = 1; }

option go_package tells the generator the import path and package name of the Go code. The form "example.test/lab/gen/common/v1;commonv1" gives both: the path, a semicolon, the package name. Make the path match where the generated files really live in your module, so imports between generated packages resolve.

Output paths: source_relative or module

Two options control where files go. paths=source_relative writes each output file next to a path mirroring the proto's location under the include root: proto/inventory/v1/inventory.proto becomes gen/inventory/v1/inventory.pb.go. module=<path> strips the module prefix from the go_package path and writes accordingly. The first is simpler to reason about and works well when the directory layout of gen matches your protos; the second helps when the go_package import path encodes a deeper layout. Pick one for the repository and keep it. How the messages themselves evolve is covered in protobuf for Go developers.

The script

set -e
mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
out=${1:-gen}
rm -rf "$out" && mkdir -p "$out"
PATH="$PWD/bin:$PATH" protoc -I proto \
  --go_out="$out" --go_opt=paths=source_relative \
  --go-grpc_out="$out" --go-grpc_opt=paths=source_relative \
  $(find proto -name '*.proto' | sort)

It takes the output directory as an argument, which is what makes the CI check possible. It deletes the target first so removed protos do not leave stale generated files, builds the plugins, and finds every .proto under proto in sorted order so the result is deterministic.

The program that uses the generated packages is short, because the point is that the imports resolve:

package main

import (
	"fmt"

	commonv1 "example.test/lab/gen/common/v1"
	inventoryv1 "example.test/lab/gen/inventory/v1"
)

func main() {
	resp := &inventoryv1.GetPriceResponse{Price: &commonv1.Money{Currency: "EUR", UnitsMinor: 1999}}
	fmt.Println(resp.GetPrice().GetCurrency(), resp.GetPrice().GetUnitsMinor())
}

Running it, and the CI check

The run script generates, shows the file list and a header, runs the program, then does the thing CI should do: regenerate into a scratch directory and compare.

set -e
bash generate.sh gen
find gen -name '*.go' | sort
echo "--- header of a generated file"
head -5 gen/inventory/v1/inventory.pb.go
echo "--- the program that uses it"
go run .
scratch=$(mktemp -d)
echo "--- CI check: regenerate into a scratch directory and compare"
bash generate.sh "$scratch/gen" && diff -r gen "$scratch/gen" && echo "generated code is up to date"
echo "--- someone edits the proto and forgets to regenerate"
sed -i.bak 's/int64 units_minor = 2;/int64 units_minor = 2;\n  string note = 3;/' proto/common/v1/money.proto
bash generate.sh "$scratch/gen"
diff -rq gen "$scratch/gen" | sed "s#$scratch/gen#fresh#" || true
mv proto/common/v1/money.proto.bak proto/common/v1/money.proto

gen/common/v1/money.pb.go
gen/inventory/v1/inventory.pb.go
gen/inventory/v1/inventory_grpc.pb.go
--- header of a generated file
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// 	protoc-gen-go v1.36.8
// 	protoc        v5.29.3
// source: inventory/v1/inventory.proto
--- the program that uses it
EUR 1999
--- CI check: regenerate into a scratch directory and compare
generated code is up to date
--- someone edits the proto and forgets to regenerate
Files gen/common/v1/money.pb.go and fresh/common/v1/money.pb.go differ

The file list shows three Go files under gen, with the directory structure mirroring the protos. The header shows what produced the file: protoc-gen-go v1.36.8 and protoc v5.29.3 (protoc 29.3 reports itself as 5.29.3 in the Go generator's comment). That header is the quickest way to debug "why did this file change": compare the versions.

The program prints EUR 1999, which proves the cross-package import works.

The CI check regenerates into a fresh directory and runs diff -r against the committed gen. For a clean tree it prints generated code is up to date. Then the script edits the proto without regenerating, as a distracted developer would, and the same comparison reports Files gen/common/v1/money.pb.go and fresh/common/v1/money.pb.go differ. In CI, that non-zero exit stops the merge. The message names the file, so the fix is obvious: run the generator and commit.

Commit generated code or generate in CI?

Both work; mixing them does not.

Commit generated code and check it in CI as above. Pros: go build and go get work with no extra tooling for consumers of your module, code review shows what changed, and you can browse it. Cons: noisy diffs, merge conflicts in generated files (resolve by regenerating, never by hand).

Generate in CI and the build, ignore gen/ in git. Pros: no diffs. Cons: every consumer of the repository needs protoc and the plugins to build, and a library module cannot be imported by others without the generated files. For libraries that other modules import, commit the generated code.

Add the go:generate directive next to the code that owns the protos, for example //go:generate bash ../scripts/generate.sh gen, so go generate ./... is the one command people remember.

Common options worth knowing

• --go-grpc_opt=require_unimplemented_servers=false removes the requirement to embed Unimplemented...Server. Do not turn it off unless you have a strong reason; the requirement is what makes adding an rpc a non-breaking change.
• --go_opt=Mfoo.proto=example.test/pkg overrides the Go import path for one file when you cannot edit it, which is useful for third-party protos.
• --go_opt=default_api_level=API_OPAQUE selects the opaque API, a newer generated style with accessor methods. Check what your protobuf-go version defaults to before switching.
• Descriptor sets (--descriptor_set_out=file.pb --include_imports) produce a binary description of your schema, used by tools, gateways and reflection.

The same thing with Buf

Buf replaces the shell script with configuration. A buf.yaml declares the modules and lint and breaking rules, and a buf.gen.yaml lists the plugins:

version: v2
managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: example.test/lab/gen
plugins:
  - local: bin/protoc-gen-go
    out: gen
    opt: paths=source_relative
  - local: bin/protoc-gen-go-grpc
    out: gen
    opt: paths=source_relative

buf generate then does what the script does, and buf lint and buf breaking check style and compatibility (Buf in CI). Buf also resolves imports from a registry, so shared protos such as the Google ones need no copying. I did not run Buf for this article, since it is a separate binary; the configuration follows its documentation. The shell script is enough for a small project, and Buf earns its place when you have many protos, many teams, or a need to enforce compatibility automatically.

The generator options are documented in the protobuf Go generated code guide and the gRPC Go quick start.