Protocol Buffers for Go Developers and Schema Evolution

If you write gRPC services you write protobuf schemas, and if you maintain them for more than a release you meet the question of what you may change without breaking the other side. The answer is a short list of rules, and the rules follow from how the format works. Learn the format once and you stop memorising the rules.

This article explains protobuf for Go developers: how a message is encoded, what the generated Go code gives you, and the schema evolution rules, each demonstrated by running old data through new schemas and the reverse. It supports the gRPC tutorial and leads into protobuf backward compatibility rules and Buf breaking change detection. It ran on Go 1.25.6 with protobuf-go v1.36.8.

How a message is encoded

A serialised message is a sequence of fields. Each field is a tag followed by a value. The tag is a varint holding (field_number << 3) | wire_type. Wire types say how to read the value: 0 is a varint (integers, bools, enums), 2 is length-delimited (strings, bytes, nested messages), 1 and 5 are fixed 64 and 32 bit values.

Take the first bytes the lab printed for the user {name: "Ada", age: 36, ...}:

0a 03 41 64 61   10 24   1a 10 ...

0a is tag (1<<3)|2: field 1, length-delimited. 03 is the length, 41 64 61 is "Ada". 10 is tag (2<<3)|0: field 2, varint, and 24 is 36. 1a is field 3, length-delimited, followed by the email. Nothing in the bytes says "name" or "age". Only the numbers.

Three consequences explain most of the rules.

1. Names do not matter on the wire. You can rename a field freely (but the JSON mapping will change, see below).
2. Numbers are forever. The number is the field's identity. Changing or reusing it changes the meaning of existing data.
3. Defaults are not sent. In proto3, a field set to its zero value is left out entirely. The reader fills in the zero value for any field it does not see.

The compactness is real: the example user took 32 bytes in protobuf and 85 bytes as JSON, and a message containing only zero values took 0 bytes.

What Go generates

A message becomes a struct with exported fields and getters, serialisation through proto.Marshal and proto.Unmarshal, reflection through ProtoReflect() and equality through proto.Equal. A few habits make using it safer.

Use the getters. u.GetName() returns "" on a nil *User; u.Name panics. Getters make nil messages harmless, which matters when an optional sub-message is absent.

Compare with proto.Equal, not == or reflect.DeepEqual. Generated structs carry internal state that makes naive comparison unreliable.

Do not copy messages by value. The structs contain a protoimpl.MessageState and go vet complains about copying them. Use proto.Clone.

Do not rely on byte-for-byte stability. Serialisation is not canonical: the same message can encode to different bytes across versions, languages and maps' order. proto.MarshalOptions{Deterministic: true} helps within one build but is not a cross-language guarantee. Never hash serialised protobuf to compare messages or build cache keys across services.

Presence: is zero set?

With plain proto3 fields, age: 0 and "no age" look the same, because the default is not sent. If the difference matters, there are three tools:

• optional: optional string nickname = 5; generates a pointer field in Go (*string), so nil means unset. The lab's v1 schema uses it for nickname and shows Nickname != nil after setting it.
• Wrapper types, google.protobuf.StringValue and friends from wrapperspb, a message that can be absent.
• A oneof when exactly one of several fields is set.

For updates, patch semantics use a field mask (google.protobuf.FieldMask) to say which fields the caller intends to change, instead of guessing from zero values. PATCH semantics has the same problem in JSON.

Unknown fields are kept

When a reader meets a tag it does not know, it does not fail and does not drop the bytes. In Go they are stored in the message's unknown fields and written back out when the message is serialised again. This is what makes rolling upgrades safe: a service in the middle that has not been updated passes along fields it has never heard of.

Proving the rules by running them

The lab generates Go for three schemas and pushes bytes between them. Version 1 of User has name, age, email, an enum plan and an optional nickname. Version 2 removes age (reserving its number and name), adds a phone field and adds an enum value PLAN_TEAM. A third file, user_bad.proto, shows the wrong ways to change the schema.

syntax = "proto3";
package user.v1;
option go_package = "example.test/lab/userv1";

enum Plan {
  PLAN_UNSPECIFIED = 0;
  PLAN_FREE = 1;
  PLAN_PRO = 2;
}

message User {
  string name = 1;
  int32 age = 2;
  string email = 3;
  Plan plan = 4;
  optional string nickname = 5;
}

syntax = "proto3";
package user.v2;
option go_package = "example.test/lab/userv2";

enum Plan {
  PLAN_UNSPECIFIED = 0;
  PLAN_FREE = 1;
  PLAN_PRO = 2;
  PLAN_TEAM = 3; // a value v1 readers have never heard of
}

// v2 removed age, kept its number and name reserved so nobody reuses it, and added a phone number.
message User {
  reserved 2;
  reserved "age";
  string name = 1;
  string email = 3;
  Plan plan = 4;
  optional string nickname = 5;
  string phone = 6;
}

syntax = "proto3";
package user.bad;
option go_package = "example.test/lab/userbad";

// The mistake: field 2 used to be an int32 age, and someone reused the number for another int32.
message Reused {
  string name = 1;
  int32 floor = 2;
  string email = 3;
}

// A different mistake: field 2 reused with a type that has a different wire type.
message Retyped {
  string name = 1;
  string nickname_text = 2;
  string email = 3;
}

//go:build tools

package main

import _ "google.golang.org/protobuf/cmd/protoc-gen-go"

package main

import (
	"encoding/hex"
	"fmt"

	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/proto"

	"example.test/lab/userbad"
	"example.test/lab/userv1"
	"example.test/lab/userv2"
)

func main() {
	nick := "ada"
	u1 := &userv1.User{Name: "Ada", Age: 36, Email: "[email protected]", Plan: userv1.Plan_PLAN_PRO, Nickname: &nick}
	b, _ := proto.Marshal(u1)
	fmt.Printf("v1 message: %d bytes: %s\n", len(b), hex.EncodeToString(b)[:36]+"...")
	j, _ := protojson.Marshal(u1)
	fmt.Printf("same data as JSON: %d bytes: %s\n", len(j), j)

	// Zero values are not sent at all.
	empty, _ := proto.Marshal(&userv1.User{Age: 0, Plan: userv1.Plan_PLAN_UNSPECIFIED})
	fmt.Printf("message with only zero values: %d bytes\n", len(empty))

	// Presence: proto3 cannot tell "age 0" from "no age", but optional can.
	fmt.Println("nickname set:", u1.Nickname != nil, "- nickname unset:", (&userv1.User{}).Nickname == nil)

	// 1. A new reader (v2) reads data written by an old writer (v1). The removed field becomes an unknown field.
	u2 := &userv2.User{}
	if err := proto.Unmarshal(b, u2); err != nil {
		panic(err)
	}
	fmt.Printf("\nv1 bytes read as v2: name=%q email=%q plan=%v phone=%q unknown bytes kept=%d\n",
		u2.Name, u2.Email, u2.Plan, u2.Phone, len(u2.ProtoReflect().GetUnknown()))

	// 2. Unknown fields survive a round trip, so a service in the middle does not destroy data it does not understand.
	b2, _ := proto.Marshal(u2)
	back := &userv1.User{}
	proto.Unmarshal(b2, back)
	fmt.Println("v1 -> v2 -> v1 round trip keeps age:", back.Age, "equal to the original:", proto.Equal(u1, back))

	// 3. An old reader (v1) reads data written by a new writer (v2) that uses a new enum value and a new field.
	w2 := &userv2.User{Name: "Grace", Plan: userv2.Plan_PLAN_TEAM, Phone: "555-0100"}
	bw, _ := proto.Marshal(w2)
	r1 := &userv1.User{}
	proto.Unmarshal(bw, r1)
	fmt.Printf("v2 bytes read as v1: name=%q plan=%v (numeric value %d), phone bytes in unknown fields=%d\n",
		r1.Name, r1.Plan, int32(r1.Plan), len(r1.ProtoReflect().GetUnknown()))

	// 4. Reusing a field number is the one change that corrupts data.
	reused := &userbad.Reused{}
	err := proto.Unmarshal(b, reused)
	fmt.Printf("\nv1 bytes read by a schema that reused field 2 as another int32: err=%v name=%q floor=%d\n", err, reused.Name, reused.Floor)
	retyped := &userbad.Retyped{}
	err = proto.Unmarshal(b, retyped)
	fmt.Printf("v1 bytes read by a schema that reused field 2 as a string: err=%v nickname_text=%q unknown bytes=%d\n",
		err, retyped.NicknameText, len(retyped.ProtoReflect().GetUnknown()))
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab proto/*.proto
go run .

v1 message: 32 bytes: 0a0341646110241a10616461406578616d70...
same data as JSON: 85 bytes: {"name":"Ada","age":36,"email":"[email protected]","plan":"PLAN_PRO","nickname":"ada"}
message with only zero values: 0 bytes
nickname set: true - nickname unset: true

v1 bytes read as v2: name="Ada" email="[email protected]" plan=PLAN_PRO phone="" unknown bytes kept=2
v1 -> v2 -> v1 round trip keeps age: 36 equal to the original: true
v2 bytes read as v1: name="Grace" plan=3 (numeric value 3), phone bytes in unknown fields=10

v1 bytes read by a schema that reused field 2 as another int32: err=<nil> name="Ada" floor=36
v1 bytes read by a schema that reused field 2 as a string: err=<nil> nickname_text="" unknown bytes=9

Old writer, new reader

The v1 bytes decoded cleanly with the v2 schema. name, email and plan came through; phone is empty because the old writer had no such field. The removed age was not lost: it sits in the unknown fields, 2 bytes (a tag and a one byte varint).

The round trip

Marshalling the v2 message and reading the result back as v1 restored age: 36, and proto.Equal said the result equals the original. Data written by a schema that knew the field survived a trip through a program that did not.

New writer, old reader

A v2 message with PLAN_TEAM and a phone number, read by v1 code, gave plan=3 (numeric value 3) and 10 bytes of unknown fields. Two lessons. In proto3, enums are open: an unrecognised value is kept as its number and does not cause an error. That means code that switches on an enum must handle unknown values with a default branch, since "Team" arriving at an old reader is normal. And the phone field is carried along untouched.

Reusing a number

The last two lines are the failure. A schema that reuses field 2 as another int32 reads the old age of 36 as floor=36, with no error. Protobuf cannot detect it; the wire types are the same, so the bytes are valid. Data is silently reinterpreted. When the reused field has a different wire type (the string version), the reader treats it as an unknown field, so nickname_text stays empty and no error is raised, and the data is dropped on the floor. Either way it is a bug that appears only when old and new data meet, which is long after the change shipped.

The cure is the reserved statement in v2: reserved 2; reserved "age";. The compiler refuses to compile a schema that defines a field with a reserved number or name. Reserve every number you remove.

The evolution rules

From the format and the experiment, the rules fall out.

Safe:

• Add a new field with a new number.
• Remove a field, and reserve its number and name.
• Add an enum value (readers must tolerate unknown values).
• Rename a field (the wire is unchanged; the JSON name and generated Go name change, so it breaks JSON users and source code).
• Change between int32, uint32, int64, uint64 and bool carefully: they share a wire type and are compatible when values fit, but a value that does not fit is truncated.

Unsafe:

• Change a field's number.
• Reuse a removed number.
• Change a field's type to one with a different wire type, or a type with a different meaning.
• Change repeated to a scalar or the reverse in anything but narrow cases.
• Move a field into or out of a oneof after the fact (some moves are compatible, most are not).
• Rename a message or package that appears in the gRPC path, or in Any type URLs.

A practical rule for teams: schemas change only by addition. When you must break, create a new package version (user.v2) and run both until the old clients are gone. The Buf tool automates the check (breaking change detection).

JSON mapping

Protobuf defines a canonical JSON form, and Go implements it in protojson. The lab output shows it: field names become lowerCamelCase by default (nickname), enums are strings ("PLAN_PRO"), and zero values are omitted unless you ask for them with EmitUnpopulated. 64-bit integers become JSON strings, because JavaScript numbers cannot hold them. Use protojson, not encoding/json, on generated messages; encoding/json ignores the protobuf rules and will produce something different. This is the format gRPC gateways and Connect speak (gRPC gateway).

Well-known types you will actually use

The protobuf distribution ships a set of common messages with Go packages under google.golang.org/protobuf/types/known. Reaching for them beats inventing your own.

• timestamppb.Timestamp is seconds and nanoseconds since the Unix epoch. Convert with timestamppb.New(t) and ts.AsTime(). Always UTC; format it at the edge.
• durationpb.Duration for lengths of time, with durationpb.New(d) and AsDuration(). Prefer it to an int64 named timeout whose unit nobody remembers.
• emptypb.Empty for rpcs that take or return nothing. A method with a response message of its own can grow fields later; Empty cannot, so use it sparingly and give most responses their own type.
• fieldmaskpb.FieldMask for partial updates, listing the paths the caller wants to change.
• structpb.Struct and Value for arbitrary JSON-shaped data, when you must carry something without a schema. It is a trap if used for things that deserve a schema.
• anypb.Any to wrap any message together with its type name. Powerful for envelopes and rich error details (error details), and it ties the wire format to type names, so renaming a message is then a breaking change.
• wrapperspb (StringValue, Int32Value) when you need "absent" as distinct from zero in places optional does not reach.

Maps, oneofs and nesting

map<string, string> fields become Go maps. Iteration order is random and so is the serialised order, which is another reason serialisation is not canonical. Map keys must be integral or string types; values can be anything except another map.

oneof declares that at most one of several fields is set. In Go it generates an interface-typed field and wrapper types (isEvent_Payload), and you switch on the concrete type. Setting one member clears the others. It models "this is either a card or a bank account" without invalid combinations. Adding a new member to an existing oneof is safe; moving an existing field into one is not.

Nested messages are pointers in Go, nil when absent. A nil nested message and an empty one are different on the wire (field absent versus a zero length value), and the getters hide the difference unless you check for nil. Check when the distinction matters, such as a "settings" message whose absence means "use the defaults" but whose presence with all zeros means "I chose zeros".

Repeated fields become slices. A nil slice and an empty one encode identically (nothing), so you cannot tell "no items" from "an empty list" across the wire. Do not build meaning on the difference.

When protobuf wins, and when it does not

The compactness shown above is a real advantage for chatty internal traffic, mobile clients on poor connections and storage of many small records. Parsing is also faster than JSON, often several times, because there is no text to scan and field names are not compared. For schemas shared across languages, the generated types and the compatibility rules are the larger benefit than the bytes.

It loses where humans need to read the data. A binary body cannot be inspected with curl; you need the schema and a tool such as buf curl or grpcurl with reflection. Logs of binary bodies are useless without decoding, so log with protojson when you must log a message (logging without leaking secrets applies to messages too: strip fields marked sensitive). And for public HTTP APIs where the consumers are unknown, JSON with an OpenAPI description is easier to adopt. Connect and gRPC-gateway exist to give protobuf services a JSON face for those callers (Connect RPC).

Measure before deciding on speed. A typical API handler spends its time in the database and on the network, and swapping JSON for protobuf moves a small slice of that. Its reliability story, the one built on the evolution rules above, is the better reason to use it.

Design habits that age well

• Enums with a zero UNSPECIFIED value as the first entry, because zero is the default and an unset field should not claim a real meaning.
• Do not use bool for something that may grow. An enum can gain a third state; a bool cannot.
• Use google.protobuf.Timestamp and Duration (timestamppb, durationpb) for times, not integers of unspecified units.
• Wrap lists in messages when you may add metadata such as a next page token.
• Keep messages small and focused, and do not reuse one message for requests and responses of different rpcs. Give each rpc its own request and response types so they can evolve independently.
• Version the package, put it in the directory structure, and never delete a released version while clients use it.
• Keep numbers 1 to 15 for the most frequent fields, since they take one byte in the tag; larger numbers cost two. This is a micro-optimisation that rarely matters, but it costs nothing at design time.

The format is specified in the protobuf encoding documentation and the compatibility guidance is in the proto best practices, both from the protobuf team. The Go API is documented at pkg.go.dev/google.golang.org/protobuf.