Protobuf Versioning and Backward Compatibility Rules

Protobuf compatibility advice usually arrives as a list of rules. Rules are easier to trust when you have watched each one hold or fail. This article takes ten common schema changes and tests each by the most direct method available: write a message with the old schema, read the same bytes with the new one, and see what the reader gets. The previous protobuf article explains the wire format and the reasoning; this one is the lookup table, with the evidence. It supports the CI check described in Buf lint and breaking change detection, and the way gRPC services carry these messages is in the gRPC tutorial. It ran on Go 1.25.6.

How the test works

The program uses protocompile, a Go library that compiles .proto text into descriptors at run time, and dynamicpb, which builds messages from descriptors. That avoids code generation, so ten schema variants can be compiled and exercised in one run. The old schema (v1) has a name, an int32 age, an email, an enum plan and a repeated tags field. A message is filled in and serialised to 31 bytes. Each variant schema then reads those bytes, and the result is printed as protobuf JSON together with the number of unknown bytes the reader held on to.

package main

import (
	"context"
	"fmt"
	"strings"

	"github.com/bufbuild/protocompile"
	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/reflect/protoreflect"
	"google.golang.org/protobuf/types/dynamicpb"
)

// compile turns .proto source text into a message descriptor, at run time, with no code generation.
func compile(src string) protoreflect.MessageDescriptor {
	c := protocompile.Compiler{Resolver: protocompile.WithStandardImports(&protocompile.SourceResolver{
		Accessor: protocompile.SourceAccessorFromMap(map[string]string{"t.proto": src}),
	})}
	files, err := c.Compile(context.Background(), "t.proto")
	if err != nil {
		panic(err)
	}
	return files[0].Messages().ByName("User")
}

const v1 = `syntax = "proto3"; package t; enum Plan { PLAN_UNSPECIFIED = 0; PLAN_FREE = 1; PLAN_PRO = 2; }
message User { string name = 1; int32 age = 2; string email = 3; Plan plan = 4; repeated string tags = 5; }`

type change struct {
	name string
	v2   string
}

func main() {
	old := compile(v1)
	// An example message written by the OLD schema.
	in := dynamicpb.NewMessage(old)
	in.Set(old.Fields().ByName("name"), protoreflect.ValueOfString("Ada"))
	in.Set(old.Fields().ByName("age"), protoreflect.ValueOfInt32(36))
	in.Set(old.Fields().ByName("email"), protoreflect.ValueOfString("[email protected]"))
	in.Set(old.Fields().ByName("plan"), protoreflect.ValueOfEnum(2))
	tags := in.Mutable(old.Fields().ByName("tags")).List()
	tags.Append(protoreflect.ValueOfString("go"))
	wire, _ := proto.Marshal(in)
	fmt.Printf("bytes written with the v1 schema: %d\n\n", len(wire))

	pre := `syntax = "proto3"; package t; enum Plan { PLAN_UNSPECIFIED = 0; PLAN_FREE = 1; PLAN_PRO = 2; }`
	changes := []change{
		{"add a field", pre + `message User { string name = 1; int32 age = 2; string email = 3; Plan plan = 4; repeated string tags = 5; string phone = 6; }`},
		{"remove a field, number reserved", pre + `message User { reserved 2; string name = 1; string email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"rename a field (same number)", pre + `message User { string name = 1; int32 years = 2; string email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"int32 age becomes int64", pre + `message User { string name = 1; int64 age = 2; string email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"int32 age becomes string", pre + `message User { string name = 1; string age = 2; string email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"reuse number 2 for a new int32", pre + `message User { string name = 1; int32 floor = 2; string email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"string email becomes bytes", pre + `message User { string name = 1; int32 age = 2; bytes email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"repeated tags becomes a single string", pre + `message User { string name = 1; int32 age = 2; string email = 3; Plan plan = 4; string tags = 5; }`},
		{"enum value removed (PLAN_PRO)", `syntax = "proto3"; package t; enum Plan { PLAN_UNSPECIFIED = 0; PLAN_FREE = 1; }
message User { string name = 1; int32 age = 2; string email = 3; Plan plan = 4; repeated string tags = 5; }`},
		{"renumber a field (email 3 to 7)", pre + `message User { string name = 1; int32 age = 2; string email = 7; Plan plan = 4; repeated string tags = 5; }`},
	}
	fmt.Printf("%-40s %s\n", "change in the new schema", "what a reader of the NEW schema gets from the v1 bytes")
	for _, c := range changes {
		md := compile(c.v2)
		out := dynamicpb.NewMessage(md)
		err := proto.Unmarshal(wire, out)
		j, _ := protojson.MarshalOptions{}.Marshal(out)
		unknown := len(out.GetUnknown())
		res := strings.ReplaceAll(string(j), " ", "")
		if err != nil {
			res = "ERROR " + err.Error()
		}
		fmt.Printf("%-40s %s  (unknown bytes: %d)\n", c.name, res, unknown)
	}
}

bytes written with the v1 schema: 31

change in the new schema                 what a reader of the NEW schema gets from the v1 bytes
add a field                              {"name":"Ada","age":36,"email":"[email protected]","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 0)
remove a field, number reserved          {"name":"Ada","email":"[email protected]","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 2)
rename a field (same number)             {"name":"Ada","years":36,"email":"[email protected]","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 0)
int32 age becomes int64                  {"name":"Ada","age":"36","email":"[email protected]","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 0)
int32 age becomes string                 {"name":"Ada","email":"[email protected]","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 2)
reuse number 2 for a new int32           {"name":"Ada","floor":36,"email":"[email protected]","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 0)
string email becomes bytes               {"name":"Ada","age":36,"email":"YWRhQGV4YW1wbGUudGVzdA==","plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 0)
repeated tags becomes a single string    {"name":"Ada","age":36,"email":"[email protected]","plan":"PLAN_PRO","tags":"go"}  (unknown bytes: 0)
enum value removed (PLAN_PRO)            {"name":"Ada","age":36,"email":"[email protected]","plan":2,"tags":["go"]}  (unknown bytes: 0)
renumber a field (email 3 to 7)          {"name":"Ada","age":36,"plan":"PLAN_PRO","tags":["go"]}  (unknown bytes: 18)

The results

Reading the table

Adding a field is the safe change. Old bytes lack it, and the reader fills in the default. This is why protobuf APIs grow by addition. Old readers of new bytes are equally fine, as the earlier article showed: they keep the field as unknown data.

Removing a field is safe only if the number is reserved. The data is parked in the unknown fields. The danger is not the removal but a later change that reuses the number, which is why the rule is reserved 2; (and reserved "age"; for the name) at the moment of removal.

A rename is invisible to the wire format. The reader got years: 36 because the number is what matters. That makes it safe for binary clients and breaking for everything that sees the name: JSON clients, generated Go code that calls GetAge, and any tool that matches on names. Treat a rename as a breaking change for anything that uses JSON, and as a source-breaking change in all languages.

Widening an integer works when the values fit. int32 to int64 is compatible because both are varints and 36 reads as 36. The reverse (64 to 32) truncates large values silently. Mixing signed and unsigned, or int32 with sint32 (which uses zigzag encoding), changes meaning even though the wire type is the same. In JSON, a 64-bit integer becomes a string, so a typed JSON client sees "36" where it had 36. The compatible-types table in the protobuf documentation lists which pairs can be swapped.

A different wire type drops the data without complaint. Changing the integer age to a string, a length-delimited type, made the reader see a field of an unexpected wire type. protobuf-go treats that as an unknown field: no error, no age, 2 bytes kept aside. A service that does this will quietly lose a column of data.

Reusing a number is the one that corrupts. The old age of 36 became floor: 36. Both are int32 varints, so everything about the bytes is valid, and nothing can detect that the meaning changed. Bugs of this kind surface weeks later in a report. This is the case reserved exists to forbid; the compiler refuses to compile a message that defines a reserved number.

string and bytes are wire-identical (both length-delimited), so changing one to the other works for valid UTF-8 data. In JSON the field changes representation: the lab shows the email as base64. And going from bytes to string can fail if the data is not valid UTF-8.

Repeated to singular loses data. Changing repeated string tags to string tags kept the last element in the lab (the data had one). With three tags, the reader would keep only the last. Going the other way (singular to repeated) is compatible for packed-capable scalars in many cases but not generally; test it.

Removing an enum value is safe on the wire but not for logic. Proto3 enums are open: the reader keeps the number 2 even though no name exists for it. Code that switches on the enum must handle unknown values. Removing a value should be done by deprecating it for a release, then reserving the number and name (reserved 2; reserved "PLAN_PRO"; inside the enum).

Renumbering is the same as removing one field and adding another. The email was lost and 18 bytes of unknown data remained: the old number 3 held the email, the new schema looked for it at 7. Never renumber. Add a new field with a new number, populate both for a while, and retire the old one.

What counts as compatible, by audience

"Compatible" depends on who is asking.

• Wire compatibility: can old bytes be read by new code and the reverse, without errors and without data loss? Most of the table.
• Source compatibility: does existing code still compile? Renames, removed fields and changed types break generated code.
• JSON compatibility: does the JSON mapping still match? Renames, type changes and enum renames break it.
• Semantic compatibility: does the data mean the same thing? Only you can judge, and it is what reuse of a number and changes in units destroy.

A change has to pass all the audiences that apply to your system. Internal services that only exchange binary protobuf care about the first; a public API with JSON clients cares about all four.

Rules to adopt

1. Never change or reuse a field number. Add a new field instead.
2. When removing a field, reserve its number and name in the same commit.
3. Do not change a field's type except to a documented compatible one, and then only widening.
4. Treat renames as breaking wherever names are visible.
5. Add enum values freely; never remove or renumber them. Make readers handle unknown values.
6. Give each rpc its own request and response messages, so they evolve independently.
7. Version packages (user.v1, user.v2) for changes that cannot be made compatibly, and run both until clients migrate.
8. Check mechanically. Run buf breaking against the main branch in CI, or write your own descriptor comparison, so a mistake cannot be merged (breaking change detection).

The official guidance is in the protobuf team's updating a message type section and the best practices, and the run-time compilation used here is from bufbuild/protocompile.