gRPC Health Checking and Reflection in Go Services

A service that is running is not necessarily a service that can take work. It may still be loading a cache, connecting to a database, or draining before a shutdown. Orchestrators, load balancers and clients all need a cheap, uniform way to ask "should I send requests here?". For HTTP services each team invents /healthz. For gRPC there is a standard: the health checking protocol, a small service defined in grpc.health.v1 that every language library implements.

Its companion is reflection, which lets a server describe its own services so tools can call them without the .proto files. This article adds both to a Go server, shows the status changes over a lifecycle, and covers how Kubernetes and clients use them. It extends graceful shutdown, and relates to the HTTP health check endpoints and to client-side load balancing. It ran on Go 1.25.6.

The health protocol

The service has two methods. Check(HealthCheckRequest) returns (HealthCheckResponse) returns the status of a named service right now. Watch returns a stream that sends the current status and then every change. The status is SERVING, NOT_SERVING, UNKNOWN or, for Watch, SERVICE_UNKNOWN. The request carries a service name, which is the fully qualified gRPC service (ping.v1.Pinger) or the empty string for the server as a whole.

grpc-go ships the implementation in google.golang.org/grpc/health. You create it, register it and set statuses:

syntax = "proto3";
package ping.v1;
option go_package = "example.test/lab/pingpb";

service Pinger {
  rpc Ping(PingRequest) returns (PingReply);
}

message PingRequest { string note = 1; }
message PingReply { string note = 1; }

//go:build tools

package main

import (
	_ "google.golang.org/grpc/cmd/protoc-gen-go-grpc"
	_ "google.golang.org/protobuf/cmd/protoc-gen-go"
)

package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"sort"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/reflection"
	reflectionpb "google.golang.org/grpc/reflection/grpc_reflection_v1"
	"google.golang.org/grpc/status"

	pb "example.test/lab/pingpb"
)

type pinger struct{ pb.UnimplementedPingerServer }

func (pinger) Ping(_ context.Context, r *pb.PingRequest) (*pb.PingReply, error) {
	return &pb.PingReply{Note: "pong " + r.GetNote()}, nil
}

func main() {
	lis, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		log.Fatal(err)
	}
	srv := grpc.NewServer()
	pb.RegisterPingerServer(srv, pinger{})

	// Health: the empty service name "" is the whole server; named entries are per service.
	hs := health.NewServer()
	healthpb.RegisterHealthServer(srv, hs)
	hs.SetServingStatus("", healthpb.HealthCheckResponse_NOT_SERVING) // not ready yet: still starting up
	hs.SetServingStatus("ping.v1.Pinger", healthpb.HealthCheckResponse_NOT_SERVING)

	// Reflection lets tools such as grpcurl discover services without the .proto files.
	reflection.Register(srv)

	go srv.Serve(lis)
	defer srv.Stop()
	conn, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	h := healthpb.NewHealthClient(conn)
	check := func(label, service string) {
		r, err := h.Check(ctx, &healthpb.HealthCheckRequest{Service: service})
		if err != nil {
			fmt.Printf("%-34s -> error %s\n", label, status.Code(err))
			return
		}
		fmt.Printf("%-34s -> %s\n", label, r.GetStatus())
	}

	check("Check overall, while starting up", "")
	check("Check ping.v1.Pinger", "ping.v1.Pinger")
	check("Check a service nobody registered", "nope.v1.Missing")

	// Watch: a stream that sends the current status and then every change.
	w, _ := h.Watch(ctx, &healthpb.HealthCheckRequest{Service: "ping.v1.Pinger"})
	first, _ := w.Recv()
	fmt.Println("Watch, first message:               ", first.GetStatus())

	hs.SetServingStatus("", healthpb.HealthCheckResponse_SERVING) // start-up finished
	hs.SetServingStatus("ping.v1.Pinger", healthpb.HealthCheckResponse_SERVING)
	next, _ := w.Recv()
	fmt.Println("Watch, after the server became ready:", next.GetStatus())
	check("Check overall, now", "")

	// A real shutdown flips to NOT_SERVING first, so load balancers stop sending new calls, then drains.
	hs.Shutdown()
	last, _ := w.Recv()
	fmt.Println("Watch, after hs.Shutdown():         ", last.GetStatus())
	check("Check overall, draining", "")

	// Reflection: ask the server what it serves.
	rc := reflectionpb.NewServerReflectionClient(conn)
	stream, _ := rc.ServerReflectionInfo(ctx)
	stream.Send(&reflectionpb.ServerReflectionRequest{MessageRequest: &reflectionpb.ServerReflectionRequest_ListServices{ListServices: ""}})
	resp, _ := stream.Recv()
	var names []string
	for _, s := range resp.GetListServicesResponse().GetService() {
		names = append(names, s.GetName())
	}
	sort.Strings(names)
	fmt.Println("reflection lists:", names)
	stream.CloseSend()
}

mkdir -p bin
go build -o bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
go build -o bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
PATH="$PWD/bin:$PATH" protoc --go_out=. --go_opt=module=example.test/lab --go-grpc_out=. --go-grpc_opt=module=example.test/lab proto/*.proto
go run .

Check overall, while starting up   -> NOT_SERVING
Check ping.v1.Pinger               -> NOT_SERVING
Check a service nobody registered  -> error NotFound
Watch, first message:                NOT_SERVING
Watch, after the server became ready: SERVING
Check overall, now                 -> SERVING
Watch, after hs.Shutdown():          NOT_SERVING
Check overall, draining            -> NOT_SERVING
reflection lists: [grpc.health.v1.Health grpc.reflection.v1.ServerReflection grpc.reflection.v1alpha.ServerReflection ping.v1.Pinger]

Reading the lifecycle

The server starts with both the whole-server entry and ping.v1.Pinger set to NOT_SERVING. This is the correct initial state for anything with start-up work: a service should not claim to be ready before it is. The first three checks show it. The overall check and the named service both say NOT_SERVING. A check for a service nobody registered returns the status code NotFound, not a status of the health enum: a probe can tell "unhealthy" from "no such service", which catches typos in probe configuration.

Then Watch is opened on ping.v1.Pinger. Its first message is the current status, NOT_SERVING. When the program sets the statuses to SERVING (start-up finished), the stream delivers the change immediately, and a plain Check of the overall status now says SERVING. This is why Watch matters for load balancers: instead of polling every few seconds, they hold one stream and react the instant a service flips.

At shutdown, hs.Shutdown() sets every registered service to NOT_SERVING and stops later SetServingStatus calls from changing it back. The watcher sees NOT_SERVING and the next check agrees. That is the first step of a graceful shutdown: tell everyone you are leaving, give them a moment to stop sending calls, then drain and stop (graceful shutdown).

What health should mean

Keep it shallow. A health check runs constantly, from many places. It should answer "can this process take a request now?", not "is the whole system healthy?". If the check calls the database and the database is slow, every instance of your service starts failing its probes at once and the orchestrator restarts all of them, turning a database blip into a full outage. Check local things (the server is up, essential start-up work is done) and, if you must check a dependency, do it in the background and report the cached result.

Distinguish liveness from readiness. Liveness answers "is this process wedged and should it be restarted?". Readiness answers "should traffic come here right now?". They need different checks. A service waiting for a dependency is not ready but not broken, and restarting it will not help. Use one service name for each (the empty name for readiness, and for example liveness registered separately), or use the process being able to answer at all as liveness.

Report per service when it helps. A server hosting several gRPC services can mark one NOT_SERVING (a feature switched off) while others work. Clients that call a specific service can check that service by name.

Flip to NOT_SERVING before stopping. Without that, load balancers keep sending requests until the connection closes. With it, new calls move away while in-flight ones finish.

Kubernetes probes

Since Kubernetes 1.24 (stable in 1.27), probes can call gRPC health services directly:

readinessProbe:
  grpc:
    port: 50051
    service: ""          # the whole server
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  grpc:
    port: 50051
    service: liveness
  periodSeconds: 10
  failureThreshold: 3

The probe calls Check; any status other than SERVING counts as a failure. Older clusters used the grpc_health_probe binary run by an exec probe; the native probe replaces it. Note that the probe does not do TLS: if your server requires TLS on that port, either serve the health service on a separate plain listener bound to localhost or the pod network, or use the exec form with TLS flags. I did not run a Kubernetes cluster for this article; the configuration follows the Kubernetes documentation.

Client-side health checking

A client with round_robin balancing can use the same protocol to avoid unhealthy backends. In the service config:

{
  "loadBalancingConfig": [{"round_robin": {}}],
  "healthCheckConfig": {"serviceName": "ping.v1.Pinger"}
}

and a blank import of google.golang.org/grpc/health in the client binary enables it. Each subchannel then watches its backend's health stream, and the picker removes backends that report NOT_SERVING. This closes the gap noted in the load balancing article, where a backend that accepts connections but cannot serve was still receiving traffic.

Reflection

Reflection answers the question that otherwise needs the .proto file: what does this server offer? reflection.Register(srv) adds a service that serves the descriptors of everything registered. Tools use it: grpcurl -plaintext localhost:50051 list prints the services, grpcurl ... describe ping.v1.Pinger prints the methods and messages, and grpcurl -d '{"note":"hi"}' ... ping.v1.Pinger/Ping calls one with JSON. Postman and other GUI tools use it for the same purpose.

The lab asks for the list through the reflection client and gets four names: grpc.health.v1.Health, two versions of the reflection service (grpc.reflection.v1 and the older v1alpha, kept for tools that have not moved) and ping.v1.Pinger. The server knows its own descriptors because the generated code registers them with the protobuf registry at import time.

Reflection is a security decision

Reflection hands anyone who can connect a complete description of your API: every method, every message, every field name. That helps developers and it helps attackers map the surface. Practical stances:

• Enable it in development and staging, where the convenience is large.
• In production, disable it, or protect it with the same authentication as everything else, or serve it only on an internal listener.
• Never rely on obscurity. An API's security must hold when the schema is known. Reflection removes a small speed bump, not a defence.

The same goes for the health service: it is harmless to expose to the network that needs it and unnecessary elsewhere. If it is on the main port with TLS and authentication interceptors, make sure your interceptor lets health checks through without credentials, or probes will fail (interceptors).

Testing

Test the lifecycle the way the lab does: a fresh server reports NOT_SERVING, flips to SERVING when start-up completes, and reports NOT_SERVING after shutdown begins. Test that the readiness check does not depend on the speed of a dependency. For reflection, assert that production configuration does not enable it, if that is your policy. The protocol is defined in the gRPC health checking guide and the Go packages are documented in pkg.go.dev/google.golang.org/grpc/health.