gRPC Metadata and Context Propagation in Go Services

A gRPC call carries a message and, beside it, metadata: a small set of key-value pairs, like HTTP headers. Authentication tokens, request ids, trace context, tenant ids and feature flags travel there. The part that causes real bugs is what happens at a hop. Service A receives a call with metadata and, to answer, calls service B. Does B see the metadata A received?

By default it does not. This is a deliberate and good design, and the cause of a stream of production mysteries: traces that break at the second service, request ids that do not appear in downstream logs, tenant headers that vanish. This article explains the model, shows it with a two-hop experiment, and gives the rules for forwarding without leaking. It builds on unary RPCs, connects to OpenTelemetry tracing and interceptors, and complements deadline propagation, which, unlike metadata, does propagate on its own through the context. It ran on Go 1.25.6.

Two sets of metadata

The Go API keeps incoming and outgoing metadata separately, both stored in the context:

• Incoming: what the caller sent. A server reads it with metadata.FromIncomingContext(ctx).
• Outgoing: what you are about to send. A client builds it with metadata.AppendToOutgoingContext(ctx, k, v, ...) or metadata.NewOutgoingContext(ctx, md).

The two sets are separate on purpose. A handler's context has the caller's metadata in the incoming slot. When it passes that context to a client call, the library looks in the outgoing slot, which is empty. So nothing is forwarded.

Contrast this with the deadline and cancellation, which live in the context itself and do propagate: pass the context and the deadline follows. And contrast it with context values (ctx.Value): they exist only in the process. A value you set with context.WithValue in A is never sent to B. If you want B to know something, it must go through the message or through metadata.

The experiment

Two services in a chain. A client sends five metadata entries to A: x-request-id, traceparent, authorization, x-tenant and a binary trace-bin. Each service answers with the metadata it saw. A calls B using one of three strategies for the outgoing context.

syntax = "proto3";
package hop.v1;
option go_package = "example.test/lab/hoppb";

service Hop {
  // Echo returns the metadata the server received, one "key=value" per line, then asks the next hop (if any) the same.
  rpc Echo(EchoRequest) returns (EchoReply);
}

message EchoRequest {}
message EchoReply { repeated string seen = 1; }

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sort"
	"strings"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/metadata"
	"google.golang.org/grpc/status"

	pb "example.test/lab/hoppb"
)

// forwardable is the allow-list of incoming metadata that may continue to the next service.
// Credentials are deliberately absent: passing the caller's token on is a decision, not a default.
var forwardable = map[string]bool{"x-request-id": true, "traceparent": true, "x-tenant": true}

// hop answers with what it received and, if it has a next hop, appends what that hop received.
type hop struct {
	pb.UnimplementedHopServer
	name    string
	next    pb.HopClient
	forward func(ctx context.Context) context.Context
}

func (h *hop) Echo(ctx context.Context, _ *pb.EchoRequest) (*pb.EchoReply, error) {
	md, _ := metadata.FromIncomingContext(ctx)
	var seen []string
	for k, v := range md {
		if strings.HasPrefix(k, ":") || k == "content-type" || k == "user-agent" || k == "grpc-accept-encoding" {
			continue // transport-level entries; not part of the story
		}
		val := strings.Join(v, ",")
		if strings.HasSuffix(k, "-bin") { // binary values are base64 on the wire; a proto string field must be valid UTF-8
			val = fmt.Sprintf("<%d binary bytes>", len(v[0]))
		}
		seen = append(seen, fmt.Sprintf("%s: %s=%s", h.name, k, val))
	}
	sort.Strings(seen)
	if h.next != nil {
		rep, err := h.next.Echo(h.forward(ctx), &pb.EchoRequest{})
		if err != nil {
			return nil, err
		}
		seen = append(seen, rep.GetSeen()...)
	}
	return &pb.EchoReply{Seen: seen}, nil
}

func start(h *hop) pb.HopClient {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	srv := grpc.NewServer()
	pb.RegisterHopServer(srv, h)
	go srv.Serve(lis)
	conn, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	return pb.NewHopClient(conn)
}

func main() {
	// "Not forwarded": the outgoing context for the next call is derived from the incoming one, which carries no metadata out.
	lazy := func(ctx context.Context) context.Context { return ctx }
	// "Allow-listed": copy only the permitted keys from the incoming metadata to the outgoing metadata.
	selective := func(ctx context.Context) context.Context {
		in, _ := metadata.FromIncomingContext(ctx)
		out := metadata.MD{}
		for k, v := range in {
			if forwardable[k] {
				out[k] = v
			}
		}
		return metadata.NewOutgoingContext(ctx, out)
	}
	// "Forward everything": what a careless blanket copy does.
	everything := func(ctx context.Context) context.Context {
		in, _ := metadata.FromIncomingContext(ctx)
		return metadata.NewOutgoingContext(ctx, in.Copy())
	}

	call := func(label string, forward func(context.Context) context.Context) {
		b := start(&hop{name: "B"})
		a := start(&hop{name: "A", next: b, forward: forward})
		ctx := metadata.Pairs(
			"X-Request-Id", "req-7", // keys are lower-cased on the way
			"traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
			"authorization", "Bearer user-token",
			"x-tenant", "acme",
			"trace-bin", string([]byte{0x00, 0x01, 0x02, 0xff}),
		)
		rep, err := a.Echo(metadata.NewOutgoingContext(context.Background(), ctx), &pb.EchoRequest{})
		fmt.Println(label)
		if err != nil {
			fmt.Println("    error:", status.Convert(err).Code(), status.Convert(err).Message())
			return
		}
		for _, s := range rep.GetSeen() {
			fmt.Println("   ", s)
		}
	}

	call("1. A calls B with the plain incoming context (nothing is forwarded)", lazy)
	call("2. A forwards only an allow-list", selective)
	call("3. A copies everything (the bearer token reaches B too)", everything)

	// Malformed metadata is rejected by the client before anything is sent.
	conn, _ := grpc.NewClient("127.0.0.1:1", grpc.WithTransportCredentials(insecure.NewCredentials()))
	ctx := metadata.AppendToOutgoingContext(context.Background(), "Bad Key", "x")
	_, err := pb.NewHopClient(conn).Echo(ctx, &pb.EchoRequest{})
	fmt.Println("\n4. a key containing a space:", status.Code(err), "-", status.Convert(err).Message())
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

1. A calls B with the plain incoming context (nothing is forwarded)
    A: authorization=Bearer user-token
    A: trace-bin=<4 binary bytes>
    A: traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    A: x-request-id=req-7
    A: x-tenant=acme
2. A forwards only an allow-list
    A: authorization=Bearer user-token
    A: trace-bin=<4 binary bytes>
    A: traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    A: x-request-id=req-7
    A: x-tenant=acme
    B: traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    B: x-request-id=req-7
    B: x-tenant=acme
3. A copies everything (the bearer token reaches B too)
    A: authorization=Bearer user-token
    A: trace-bin=<4 binary bytes>
    A: traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    A: x-request-id=req-7
    A: x-tenant=acme
    B: authorization=Bearer user-token
    B: trace-bin=<4 binary bytes>
    B: traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    B: x-request-id=req-7
    B: x-tenant=acme

4. a key containing a space: Internal - header key "bad key" contains illegal characters not in [0-9a-z-_.]

Strategy 1: pass the context as is

A received all five entries. B received none. The B: lines are missing from the first result. Nothing was forwarded, because the outgoing metadata was empty. This is what happens in code that just does next.Echo(ctx, req) inside a handler, and it is why traces break: the traceparent header never reaches B, so B starts a new trace.

Also notice that the key X-Request-Id, sent in mixed case, arrived at A as x-request-id. gRPC metadata keys are lower-cased by the library and must contain only lowercase letters, digits, -, _ and .. The last line of the output shows what happens otherwise: a key with a space is refused on the client before anything is sent.

Strategy 2: forward an allow-list

The selective function reads the incoming metadata and builds an outgoing set containing only keys in an explicit allow-list: x-request-id, traceparent, x-tenant. B received exactly those three. The token and the binary value stayed at A.

This is the strategy to use. Request ids and trace context should flow through every hop so you can follow one request across services (request IDs). Tenant or locale headers that every service needs go on the list. Everything else is a decision.

Strategy 3: copy everything

The everything function copies all incoming metadata to the outgoing context. It looks convenient, and the output shows the problem: B received authorization=Bearer user-token. The user's token, meant for A, was passed to a service that never asked for it and may log it, forward it somewhere else or trust it as proof of the user's identity.

This is the confused deputy problem. A holds a credential issued for one audience and hands it to another. Whether B should receive it is a design question: sometimes the answer is yes (B acts on the user's behalf and verifies the token's audience), often no (B should authenticate A as a service and receive the user's identity in a controlled field). Never decide by default; decide per hop, in code that someone reviewed.

Binary values

Metadata values are ASCII strings, except for keys that end in -bin. For those, the library base64-encodes the value on the way out and decodes it on arrival, so the handler sees the raw bytes. The lab sent four bytes (including 0x00 and 0xff) in trace-bin and the handler saw four bytes. Use -bin for anything that is not printable text: serialised protobuf, hashes, signatures. A binary value in an ordinary key is a protocol violation and fails in confusing ways.

Limits and reserved names

• Size. Servers limit the total size of incoming metadata (about 8 KiB by default in grpc-go, grpc.MaxHeaderListSize changes it), and proxies in between have their own limits. Large values belong in the message.
• Reserved prefix. Keys starting with grpc- are reserved for the protocol (grpc-timeout, grpc-status, grpc-encoding). Do not use that prefix. Pseudo-headers starting with : are HTTP/2's.
• Multiple values. A key can have several values; md.Get(k) returns a slice. Decide whether you want the first, the last or all.
• Order. The order of different keys is not significant.

Forwarding in one place

Writing the allow-list in every handler invites mistakes. Do it once, in a client interceptor that runs on every outgoing call: it reads the incoming metadata from the context, builds the outgoing set from the allow-list and calls the invoker. Every call from every handler is then covered, and the list is in one reviewed file:

func forwardMetadata(allow ...string) grpc.UnaryClientInterceptor {
	return func(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
		if in, ok := metadata.FromIncomingContext(ctx); ok {
			for _, k := range allow {
				if v := in.Get(k); len(v) > 0 {
					ctx = metadata.AppendToOutgoingContext(ctx, append([]string{k}, v...)...)
				}
			}
		}
		return invoker(ctx, method, req, reply, cc, opts...)
	}
}

Write the stream version too. Pair it with a server interceptor that puts the request id in the context for logging, generating one when the caller sent none (interceptors). With OpenTelemetry, you rarely forward traceparent by hand: the otelgrpc stats handlers inject and extract it, which is one more reason to use them (gRPC and OpenTelemetry).

What not to put in metadata

Business data. If the handler cannot do its job without it, it belongs in the message, where it is typed, versioned and validated.

Large or sensitive blobs. Metadata is logged by proxies and tools more often than message bodies.

Anything you would not want a downstream to see. Treat the outgoing metadata as an API of its own, with a documented list of keys and who may set them.

Identity claims from callers. A header like x-user-id that any caller can set is not an identity. Take identity from verified credentials (mTLS or a validated token) and, if you forward it, forward the verified value in a key that your edge strips from external requests (authentication).

Server to client: headers and trailers

The flow in the other direction uses grpc.SetHeader, grpc.SendHeader and grpc.SetTrailer on the server, and grpc.Header(&md) and grpc.Trailer(&md) call options on the client, as shown in unary RPCs. The same rules apply to what A returns to its caller: do not copy B's response metadata to the client wholesale.

Testing

Test propagation the way the lab does: a chain of in-process services where each reports what it received, and assertions about presence and absence. Assert that the request id and trace context arrive at the last hop, and, just as important, that the authorisation header does not. Run that as a regression test, since the failure mode of both a missing and an extra forward is silent. The Go API is in the metadata package and the rules for keys and binary values in the gRPC metadata guide.