Build a Microservice Pair in Go Connected by gRPC Calls

One service is a program. Two services that call each other are a distributed system, with all the new ways to fail that implies: the second service is slow, or down, or answers after you stopped waiting; the call succeeds but the response is lost, so you call again; the first step succeeds and the second fails, leaving something half done. Most advice about microservices is about avoiding those problems by not splitting. When you do split, a handful of small decisions decide whether the pair is reliable.

This project builds the smallest realistic pair in Go: an orders service that takes an order, and an inventory service that owns stock. They talk over gRPC. The code is short, and each decision in it corresponds to a failure that the program then provokes. It uses ideas from the rest of this series: deadlines, retries, error codes, metadata, health checking and graceful shutdown. It ran on Go 1.25.6.

The contract

One .proto file defines both services here for compactness. In a real system they live in separate files (and usually separate repositories) so each team owns its contract and the packages version independently.

syntax = "proto3";

package shop.v1;

option go_package = "example.test/lab/shoppb";

// Inventory owns stock levels.
service InventoryService {
  // Reserve holds stock for an order. Repeating a call with the same reservation_id changes nothing.
  rpc Reserve(ReserveRequest) returns (ReserveResponse);
  // Release returns a reservation to stock (used to undo a reservation when a later step fails).
  rpc Release(ReleaseRequest) returns (ReleaseResponse);
}

message ReserveRequest {
  string reservation_id = 1;
  string sku = 2;
  int32 quantity = 3;
}
message ReserveResponse { int32 remaining = 1; }
message ReleaseRequest { string reservation_id = 1; }
message ReleaseResponse {}

// Orders owns orders and calls Inventory.
service OrderService {
  rpc PlaceOrder(PlaceOrderRequest) returns (PlaceOrderResponse);
}

message PlaceOrderRequest {
  string order_id = 1;
  string sku = 2;
  int32 quantity = 3;
  int64 payment_cents = 4;
}
message PlaceOrderResponse {
  string order_id = 1;
  string status = 2;
}

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

Two design choices are worth pointing at.

reservation_id makes Reserve idempotent. The caller chooses an id, and calling Reserve again with the same id does not take stock twice. This is the property that makes retries safe. Without it, a retry after a lost response would reserve twice, and the system would slowly leak stock. The id should be derived from something stable and unique to the operation; here it is res- plus the order id.

Release exists because reservation is the first step of a sequence, and a later step may fail. A service that has a "do" without an "undo" cannot recover from partial failure.

Inventory

package main

import (
	"context"
	"sync"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	pb "example.test/lab/shoppb"
)

// inventory is a complete service: in-memory stock, idempotent reservations.
type inventory struct {
	pb.UnimplementedInventoryServiceServer
	mu           sync.Mutex
	stock        map[string]int32
	reservations map[string]*pb.ReserveRequest
	calls        int
}

func newInventory() *inventory {
	return &inventory{stock: map[string]int32{"keyboard": 5, "mouse": 1}, reservations: map[string]*pb.ReserveRequest{}}
}

func (s *inventory) Reserve(ctx context.Context, r *pb.ReserveRequest) (*pb.ReserveResponse, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.calls++
	if r.GetReservationId() == "" || r.GetQuantity() <= 0 {
		return nil, status.Error(codes.InvalidArgument, "reservation_id and a positive quantity are required")
	}
	if _, done := s.reservations[r.GetReservationId()]; done { // a retry of a call that already succeeded
		return &pb.ReserveResponse{Remaining: s.stock[r.GetSku()]}, nil
	}
	have, known := s.stock[r.GetSku()]
	if !known {
		return nil, status.Errorf(codes.NotFound, "unknown sku %q", r.GetSku())
	}
	if have < r.GetQuantity() {
		return nil, status.Errorf(codes.FailedPrecondition, "only %d of %q left", have, r.GetSku())
	}
	s.stock[r.GetSku()] = have - r.GetQuantity()
	s.reservations[r.GetReservationId()] = r
	return &pb.ReserveResponse{Remaining: s.stock[r.GetSku()]}, nil
}

func (s *inventory) Release(ctx context.Context, r *pb.ReleaseRequest) (*pb.ReleaseResponse, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	if res, ok := s.reservations[r.GetReservationId()]; ok {
		s.stock[res.GetSku()] += res.GetQuantity()
		delete(s.reservations, r.GetReservationId())
	}
	return &pb.ReleaseResponse{}, nil // releasing something already released is fine
}

It is a complete service in 50 lines. Reserve validates the input and returns InvalidArgument; looks the reservation id up first and returns success without changing anything if it was already done; returns NotFound for an unknown item and FailedPrecondition when stock is too low (the request is fine, the state does not allow it); and otherwise takes the stock and records the reservation. Release is also idempotent: releasing a reservation that does not exist is not an error, because the caller's goal (this reservation does not hold stock) is already true.

The lock covers the whole check-and-update, which is what makes the stock check correct under concurrent calls. A real implementation would persist reservations and stock in a database with the same logic in a transaction, and the unique reservation id would be a primary key.

Orders

package main

import (
	"context"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/metadata"
	"google.golang.org/grpc/status"

	pb "example.test/lab/shoppb"
)

// orders calls inventory. Its job is orchestration: reserve, charge, and undo the reservation if charging fails.
type orders struct {
	pb.UnimplementedOrderServiceServer
	inv     pb.InventoryServiceClient
	charge  func(cents int64) error // stands in for a payment provider
	timeout time.Duration
}

func (o *orders) PlaceOrder(ctx context.Context, r *pb.PlaceOrderRequest) (*pb.PlaceOrderResponse, error) {
	if r.GetOrderId() == "" {
		return nil, status.Error(codes.InvalidArgument, "order_id is required")
	}
	// Each downstream call gets a slice of the caller's remaining budget, never a fresh unbounded context.
	callCtx, cancel := context.WithTimeout(ctx, o.timeout)
	defer cancel()

	_, err := o.inv.Reserve(callCtx, &pb.ReserveRequest{ReservationId: "res-" + r.GetOrderId(), Sku: r.GetSku(), Quantity: r.GetQuantity()})
	if err != nil {
		// Translate at the boundary: the caller of orders should not need to know inventory exists.
		switch status.Code(err) {
		case codes.FailedPrecondition:
			return nil, status.Error(codes.FailedPrecondition, "not enough stock: "+status.Convert(err).Message())
		case codes.NotFound, codes.InvalidArgument:
			return nil, status.Error(codes.InvalidArgument, "bad item: "+status.Convert(err).Message())
		case codes.Unavailable, codes.DeadlineExceeded:
			return nil, status.Error(codes.Unavailable, "stock service is not responding, try again")
		}
		return nil, status.Error(codes.Internal, "internal error")
	}

	if err := o.charge(r.GetPaymentCents()); err != nil {
		// The reservation must not stay held. Use a fresh short context: the caller's may already be cancelled.
		undo, undoCancel := context.WithTimeout(context.WithoutCancel(ctx), time.Second)
		defer undoCancel()
		o.inv.Release(undo, &pb.ReleaseRequest{ReservationId: "res-" + r.GetOrderId()})
		return nil, status.Error(codes.FailedPrecondition, "payment declined")
	}
	return &pb.PlaceOrderResponse{OrderId: r.GetOrderId(), Status: "confirmed"}, nil
}

// propagate is a client interceptor that forwards the request id so both services log the same one.
func propagate(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
	if md, ok := metadata.FromIncomingContext(ctx); ok {
		if v := md.Get("x-request-id"); len(v) > 0 {
			ctx = metadata.AppendToOutgoingContext(ctx, "x-request-id", v[0])
		}
	}
	return invoker(ctx, method, req, reply, cc, opts...)
}

PlaceOrder is orchestration: validate, reserve, charge, and undo the reservation if charging fails. Four details deserve attention.

A bounded context for the downstream call. context.WithTimeout(ctx, o.timeout) derives from the caller's context, so the caller's deadline still applies and the call is also limited to 500 ms of its own. The inventory call can never run longer than either. It is also never context.Background(), which would lose the caller's deadline (deadlines).

Translating errors at the boundary. Inventory returns its vocabulary: NotFound for an unknown sku, FailedPrecondition for low stock. Orders turns them into its own: InvalidArgument: bad item and FailedPrecondition: not enough stock. It maps Unavailable and DeadlineExceeded from downstream to Unavailable for its own caller with a message that does not name the other service, and anything unrecognised to Internal. If orders passed inventory's errors through unchanged, its callers would depend on inventory's behaviour and could confuse "the order's item is unknown" with "the order does not exist".

Compensation. If the charge fails after the reservation succeeded, the code calls Release. It uses context.WithoutCancel(ctx) with a one second timeout of its own, because the incoming context may already be cancelled (the client could have given up), and cancellation of the caller must not leave stock held. This is one of the few places where detaching from the caller's context is right, and it is deliberate and commented.

A client interceptor that forwards x-request-id from the incoming metadata to the outgoing call, so a request can be followed through both services' logs (metadata propagation). The allow-list is a single key.

Wiring it together

The main function starts inventory (with a health service), builds the orders client with a retry policy for Reserve on UNAVAILABLE, starts orders, and runs scenarios through a public client. The retry configuration is JSON in the default service config:

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"net"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/status"

	pb "example.test/lab/shoppb"
)

func listen() net.Listener {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	return lis
}

func dial(addr string, opts ...grpc.DialOption) *grpc.ClientConn {
	conn, err := grpc.NewClient(addr, append([]grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}, opts...)...)
	if err != nil {
		log.Fatal(err)
	}
	return conn
}

const retryConfig = `{"methodConfig":[{"name":[{"service":"shop.v1.InventoryService","method":"Reserve"}],
 "retryPolicy":{"maxAttempts":3,"initialBackoff":"0.01s","maxBackoff":"0.05s","backoffMultiplier":2,"retryableStatusCodes":["UNAVAILABLE"]}}]}`

func main() {
	// Service 1: inventory, with the standard health service.
	invLis := listen()
	inv := newInventory()
	invSrv := grpc.NewServer()
	pb.RegisterInventoryServiceServer(invSrv, inv)
	hs := health.NewServer()
	healthpb.RegisterHealthServer(invSrv, hs)
	go invSrv.Serve(invLis)

	// Service 2: orders, with a client to inventory that has a retry policy, a request-id interceptor and a per-call timeout.
	paymentOK := true
	ord := &orders{
		inv: pb.NewInventoryServiceClient(dial(invLis.Addr().String(), grpc.WithDefaultServiceConfig(retryConfig), grpc.WithChainUnaryInterceptor(propagate))),
		charge: func(int64) error {
			if paymentOK {
				return nil
			}
			return errors.New("card declined")
		},
		timeout: 500 * time.Millisecond,
	}
	ordLis := listen()
	ordSrv := grpc.NewServer()
	pb.RegisterOrderServiceServer(ordSrv, ord)
	go ordSrv.Serve(ordLis)
	defer ordSrv.Stop()

	// A caller of the public service.
	client := pb.NewOrderServiceClient(dial(ordLis.Addr().String()))
	place := func(label string, id, sku string, qty int32) {
		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		defer cancel()
		r, err := client.PlaceOrder(ctx, &pb.PlaceOrderRequest{OrderId: id, Sku: sku, Quantity: qty, PaymentCents: 1999})
		if err != nil {
			fmt.Printf("%-44s -> %s: %s\n", label, status.Code(err), status.Convert(err).Message())
			return
		}
		fmt.Printf("%-44s -> OK %s\n", label, r.GetStatus())
	}
	stock := func() string {
		inv.mu.Lock()
		defer inv.mu.Unlock()
		return fmt.Sprintf("keyboard=%d mouse=%d", inv.stock["keyboard"], inv.stock["mouse"])
	}

	place("order 1: 2 keyboards", "o1", "keyboard", 2)
	fmt.Println("    stock:", stock())
	place("order 1 again (a client retry)", "o1", "keyboard", 2)
	fmt.Println("    stock:", stock(), "(unchanged: the reservation id made the retry harmless)")
	place("order 2: 2 mice, only 1 left", "o2", "mouse", 2)
	place("order 3: an unknown item", "o3", "teapot", 1)
	place("order 4: missing order id", "", "mouse", 1)

	paymentOK = false
	place("order 5: payment declined", "o5", "mouse", 1)
	fmt.Println("    stock:", stock(), "(the reservation was released)")
	paymentOK = true

	// Take inventory down while orders keeps running.
	invSrv.Stop()
	place("order 6: inventory is down", "o6", "keyboard", 1)
	fmt.Println("    reserve calls that reached inventory in total:", inv.calls)
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

order 1: 2 keyboards                         -> OK confirmed
    stock: keyboard=3 mouse=1
order 1 again (a client retry)               -> OK confirmed
    stock: keyboard=3 mouse=1 (unchanged: the reservation id made the retry harmless)
order 2: 2 mice, only 1 left                 -> FailedPrecondition: not enough stock: only 1 of "mouse" left
order 3: an unknown item                     -> InvalidArgument: bad item: unknown sku "teapot"
order 4: missing order id                    -> InvalidArgument: order_id is required
order 5: payment declined                    -> FailedPrecondition: payment declined
    stock: keyboard=3 mouse=1 (the reservation was released)
order 6: inventory is down                   -> Unavailable: stock service is not responding, try again
    reserve calls that reached inventory in total: 5

What the scenarios show

The happy path and a retry. Order o1 for two keyboards succeeded and left keyboard=3. Sending the same order again (what a client does when it times out and retries) also succeeded and left stock at keyboard=3. Orders called Reserve with res-o1 the second time, and inventory recognised it. That is idempotency at the service boundary doing its job.

Business errors. Two mice when one remained returned FailedPrecondition: not enough stock: only 1 of "mouse" left. An unknown item returned InvalidArgument. A missing order id returned InvalidArgument without calling inventory, which is why the final count of reserve calls does not include it. Each failure has a distinct code the client can act on.

Compensation. Order o5 reserved the last mouse, the payment function declined, and orders released the reservation. The output shows mouse=1 again afterwards and the client got FailedPrecondition: payment declined. Without the release, the last mouse would be locked in an abandoned reservation.

Outage. The program then stopped inventory. Order o6 returned Unavailable: stock service is not responding, try again. The orders client's retry policy tried the Reserve call up to three times against the dead service before giving up (the failed attempts never reached inventory, so its counter, 5 reserve calls in total, includes only the earlier orders). The caller's message is clean, and nothing in it names inventory.

Where the boundary should be

The easy mistake is splitting by noun and discovering that every request needs both halves. Orders and inventory are a reasonable split because each owns data that has its own consistency rules (stock cannot go negative; an order has a lifecycle), and the call between them is narrow: reserve, release. If every orders operation also needed inventory's other data, or if the two always deployed together, they would be one service with two packages.

Questions to ask before splitting:

• Who owns the data? Each piece of data has exactly one owning service. Others ask for it through the API and do not read the owner's tables. Shared databases between services give you the coupling of a monolith with the failure modes of a distributed system.
• What happens when the other service is down? If the answer is "we cannot do anything useful", the dependency is hard, and its availability caps yours. Two services each at 99.9 percent give about 99.8 percent for the combination (SLOs). Consider whether a degraded mode exists: accept the order as pending and confirm later.
• Do the teams and release cycles differ? The strongest reason to split is that different people need to change and deploy parts independently. If one team owns both, a modular monolith with a clean internal interface gives most of the benefit and none of the network.

Synchronous calls or events

The pair above uses synchronous gRPC calls: orders waits for inventory. That is simple and gives an immediate answer, which suits a user waiting for a confirmation. It also means orders fails when inventory fails, as the outage scenario showed.

The alternative is asynchronous messaging: orders records the order as pending, publishes an OrderPlaced event, inventory consumes it, reserves, and publishes StockReserved or StockRejected, and orders updates the order. Orders no longer depends on inventory being up at that moment. The price is complexity: the user does not get an immediate answer, there are more states to model, messages can be delivered more than once (so consumers must be idempotent, the same property the reservation id gives here), and ordering and failure handling live in the message system. A notification service shows the queue side in detail.

A pragmatic route is to start synchronous and keep the interface narrow and idempotent. If it later has to become asynchronous, Reserve with a reservation id is already shaped like a message handler.

The saga in outline

The compensation in PlaceOrder is a miniature saga: a sequence of local steps, each with an undo, run so that if one fails the earlier ones are compensated. A fuller one has more steps (reserve stock, charge payment, create shipment) and the same structure:

type step struct {
	name string
	do   func(ctx context.Context) error
	undo func(ctx context.Context) // must be idempotent and safe to retry
}

func run(ctx context.Context, steps []step) error {
	var done []step
	for _, s := range steps {
		if err := s.do(ctx); err != nil {
			undoCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
			defer cancel()
			for i := len(done) - 1; i >= 0; i-- {
				done[i].undo(undoCtx)
			}
			return err
		}
		done = append(done, s)
	}
	return nil
}

Two properties make it work. Every step is idempotent, so a repeat after a lost response is harmless. And every undo is too, and is safe to run for a step that never completed. What this sketch lacks is durability: if the process dies halfway, nothing remembers which steps ran. Production sagas persist their progress (a state row per order, or a workflow engine such as Temporal) and resume after a crash. Start with the sketch plus expiring reservations, and adopt a workflow engine when the number of steps and failure modes justifies it.

Running the pair locally

For development, each service is a binary and a compose.yaml runs them together with their databases:

services:
  inventory:
    build: ./inventory
    ports: ["50051:50051"]
  orders:
    build: ./orders
    environment:
      INVENTORY_ADDR: inventory:50051
    ports: ["50052:50052"]
    depends_on: [inventory]

Take addresses from environment variables, never hard-code them. Use the dns:/// scheme and round_robin when there are several replicas. Bring up the stack, call orders with grpcurl (reflection enabled in development only), and run the same scenarios as the lab against it. A scripted run against the real containers catches the configuration problems that in-process tests cannot: wrong ports, missing environment, a service that starts before its dependency is ready.

What this version leaves out

This is the core, not a production system. The pieces to add, in the order I would add them:

1. Persistence. State in a database, with reservation ids as unique keys. Orders must record its own state too (reserved, confirmed, failed), so a crash between steps is recoverable.
2. Crash safety in the sequence. If orders crashes after Reserve and before Release, nobody releases. Solutions: reservations that expire unless confirmed (a TTL), a reconciliation job that releases reservations for orders that never confirmed, or a saga with a durable log. Use the simplest that fits; expiring reservations is often enough.
3. Authentication between services. mTLS and per-method authorisation (authentication).
4. Observability. The OpenTelemetry stats handlers on both servers and the client, so one trace spans both services (tracing), plus metrics by method and code.
5. Deadlines at the edge. A default deadline on incoming calls to orders, so no request runs unbounded.
6. Real payments, which have their own idempotency keys (idempotency).
7. Deployment. Readiness checks that include "can reach inventory" only if you can tolerate the cascading effect; usually readiness should not depend on downstream health (health checks). Graceful shutdown on both services.
8. Load balancing between inventory replicas (load balancing).

Reading the code as a checklist

The orders service shows five habits that are worth applying to any service that calls another: every outbound call has a deadline derived from the inbound one, every outbound call is safe to repeat, every downstream error is translated before it leaves, every multi-step operation has a way to undo, and every inbound input is validated before any outbound call is made. When reviewing a pull request that adds a new downstream call, ask those five questions in that order. Most incidents in service graphs trace back to one of them being skipped.

Testing the pair

Test each service alone and the pair together. Inventory is tested by calling it through bufconn, with a table of cases (enough stock, not enough, unknown, repeated reservation id, release twice) and a concurrency test where many goroutines reserve the last unit and exactly one wins (testing gRPC services). Orders is tested against a fake inventory server that returns scripted statuses, so the translation table, the compensation path and the retry behaviour are exercised without the real service. The end-to-end scenario above, with both real services in one process, is the smoke test: it belongs in CI. Add a fault-injection test that makes inventory slow, so the 500 ms bound and the caller's deadline are checked, and one that stops it, as o6 does.

The gRPC project's basics tutorial covers the call mechanics used here, and the saga and outbox patterns that follow naturally are described in the microservices.io pattern catalogue.