Health Check and Readiness Endpoints in Go Services

Most services have a /health endpoint, and most of them are wrong in one of two directions. Either they return 200 OK unconditionally, which tells the orchestrator nothing, or they ping the database, the cache and three downstream APIs, so one slow dependency takes every instance out of rotation, and a slow query turns into a restart loop. The fix is to stop treating health as one question.

This article splits it into the three questions an orchestrator asks, builds the handlers with bounded and parallel checks, and ties readiness to shutdown. It belongs to the web cluster that ends in a production HTTP server, and the shutdown half is the same sequence as in graceful shutdown. Everything ran on Go 1.25.6 with the standard library only.

Three endpoints, three questions

Liveness: is this process stuck beyond recovery? If the answer is no, the platform kills and restarts the container. The check should be as dumb as possible, a handler that returns 200 proves the HTTP server is accepting connections and a goroutine is scheduled. It must not touch anything outside the process. If liveness fails when the database is down, a database outage becomes a fleet-wide restart storm, which makes the outage worse and slower to recover from.

Readiness: can this instance serve a request right now? If not, the platform stops sending it traffic but leaves it running. This is where dependencies belong, restricted to the ones a typical request cannot succeed without. A cache you can bypass does not belong in readiness; a primary database usually does. An instance that is warming up, draining or temporarily overloaded reports itself not ready and recovers on its own.

Startup: has the slow initialization finished? Kubernetes holds the other probes back until this one passes, which gives a service that loads a large model or runs migrations a long first deadline without loosening the liveness settings for the rest of its life. If startup is quick, skip it.

The Kubernetes documentation on probes describes how each is used, and the differences in failure behavior (restart versus remove from endpoints) are the reason for the split.

Bounded checks, in parallel

A readiness check calls out to other systems, and the call can hang. The probe has its own timeout (usually a second or two), so your handler must finish well inside that. Run each check with its own context deadline, run them concurrently, and report each result separately.

package main

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"sync"
	"sync/atomic"
	"time"
)

type Check func(ctx context.Context) error

type Health struct {
	checks   map[string]Check
	timeout  time.Duration
	draining atomic.Bool

	mu       sync.Mutex
	cachedAt time.Time
	cached   map[string]string
	ttl      time.Duration
	runs     atomic.Int64
}

func (h *Health) results(ctx context.Context) map[string]string {
	h.mu.Lock()
	defer h.mu.Unlock()
	if h.cached != nil && time.Since(h.cachedAt) < h.ttl {
		return h.cached // a probe every second must not become a query every second
	}
	h.runs.Add(1)
	out := make(map[string]string, len(h.checks))
	var wg sync.WaitGroup
	var rm sync.Mutex
	for name, check := range h.checks {
		wg.Add(1)
		go func() {
			defer wg.Done()
			cctx, cancel := context.WithTimeout(ctx, h.timeout)
			defer cancel()
			status := "ok"
			if err := check(cctx); err != nil {
				if errors.Is(err, context.DeadlineExceeded) {
					status = "timeout"
				} else {
					status = "fail: " + err.Error()
				}
			}
			rm.Lock()
			out[name] = status
			rm.Unlock()
		}()
	}
	wg.Wait()
	h.cached, h.cachedAt = out, time.Now()
	return out
}

func (h *Health) Live(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK) // nothing outside this process is consulted
	io.WriteString(w, "ok\n")
}

func (h *Health) Ready(w http.ResponseWriter, r *http.Request) {
	status := http.StatusOK
	body := map[string]any{}
	if h.draining.Load() {
		status = http.StatusServiceUnavailable
		body["draining"] = true
	}
	checks := h.results(r.Context())
	for _, v := range checks {
		if v != "ok" {
			status = http.StatusServiceUnavailable
		}
	}
	body["checks"] = checks
	w.Header().Set("Content-Type", "application/json")
	w.Header().Set("Cache-Control", "no-store")
	w.WriteHeader(status)
	json.NewEncoder(w).Encode(body)
}

func main() {
	var dbDown atomic.Bool
	var cacheHang atomic.Bool
	h := &Health{
		timeout: 100 * time.Millisecond,
		ttl:     time.Second,
		checks: map[string]Check{
			"database": func(ctx context.Context) error {
				if dbDown.Load() {
					return errors.New("connection refused")
				}
				return nil
			},
			"cache": func(ctx context.Context) error {
				if cacheHang.Load() {
					<-ctx.Done() // a dependency that never answers
					return ctx.Err()
				}
				return nil
			},
		},
	}
	mux := http.NewServeMux()
	mux.HandleFunc("GET /livez", h.Live)
	mux.HandleFunc("GET /readyz", h.Ready)
	srv := httptest.NewServer(mux)
	defer srv.Close()

	probe := func(label, path string) {
		start := time.Now()
		res, err := http.Get(srv.URL + path)
		if err != nil {
			panic(err)
		}
		defer res.Body.Close()
		b, _ := io.ReadAll(res.Body)
		fmt.Printf("%-28s %s -> %d %s (%s)\n", label, path, res.StatusCode, trim(b), time.Since(start).Round(100*time.Millisecond))
	}
	expire := func() { time.Sleep(1100 * time.Millisecond) } // let the cached result age out

	probe("healthy", "/readyz")
	dbDown.Store(true)
	probe("database down, cached", "/readyz")
	expire()
	probe("database down, fresh", "/readyz")
	probe("liveness while db is down", "/livez")
	dbDown.Store(false)
	cacheHang.Store(true)
	expire()
	probe("cache hangs", "/readyz")
	cacheHang.Store(false)
	expire()
	h.draining.Store(true)
	probe("draining before shutdown", "/readyz")
	fmt.Println("check runs executed:", h.runs.Load())
}

func trim(b []byte) string {
	s := string(b)
	for len(s) > 0 && (s[len(s)-1] == '\n') {
		s = s[:len(s)-1]
	}
	return s
}

healthy                      /readyz -> 200 {"checks":{"cache":"ok","database":"ok"}} (0s)
database down, cached        /readyz -> 200 {"checks":{"cache":"ok","database":"ok"}} (0s)
database down, fresh         /readyz -> 503 {"checks":{"cache":"ok","database":"fail: connection refused"}} (0s)
liveness while db is down    /livez -> 200 ok (0s)
cache hangs                  /readyz -> 503 {"checks":{"cache":"timeout","database":"ok"}} (100ms)
draining before shutdown     /readyz -> 503 {"checks":{"cache":"ok","database":"ok"},"draining":true} (0s)
check runs executed: 4

Notice what the output shows. A hung cache cost one timeout entry and about 100 ms, not a hung probe, because the check's context expired and the others had already finished in parallel. The second request with the database down still reported healthy for a moment, because it was served from the one second cache; the line after it, taken after the entry aged out, reports fail: connection refused and a 503. Liveness stayed 200 throughout, which is the point: a database outage should show up as an instance that is not ready, never as a process that needs killing. The count at the end shows how few times the checks ran for six probes: four, not six, because two of them landed inside a cache window.

The cache is a trade-off, and I want to be plain about it. With a one second TTL, readiness can lag reality by up to a second. For a probe that fires every five seconds that is irrelevant, and it protects the database from an overzealous platform that probes every instance every second from several places. If you need freshness, shorten the TTL; do not remove it.

What belongs in a check

Keep each one cheap and specific. A database check is db.PingContext(ctx) or SELECT 1, never a business query. A cache check is a PING. A downstream HTTP API gets a HEAD or a dedicated /status call with a tight timeout, and think twice about whether you should check it at all: if that API being down means you can still serve most requests, its absence from readiness is a feature. Include a check for something the instance owns, such as "migrations applied" or "configuration loaded", because those are the failures a restart actually fixes.

Return enough detail to debug and not enough to attack. The JSON above names the failing check and a short reason, which helps the engineer holding curl. It should not echo connection strings or driver errors that contain hostnames. If the endpoints are reachable from the internet, serve them on a separate internal port, or at least strip the body for requests that do not come from the platform.

Readiness and shutdown

When a deploy sends SIGTERM, the instance is still in the load balancer's list for a few seconds. If it closes its listener immediately, requests routed in that window fail. The sequence that avoids it is: flip readiness to failing (draining.Store(true) above), wait a few seconds for the platform to remove the instance from rotation (the length of the probe period times the failure threshold, plus propagation), then call Server.Shutdown. The graceful shutdown article builds the whole thing, and the HTTP details cover the connection side. Liveness stays green throughout, so the platform does not kill the pod mid-drain.

The probe configuration

On Kubernetes the matching configuration is short. The numbers are a starting point, not a recommendation for every service:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2
  timeoutSeconds: 2

The liveness check tolerates 30 seconds of failure before a restart, which is generous on purpose, because killing a process is the most disruptive action available. Readiness reacts in about ten seconds. Tune both from observed behavior (the Google SRE book chapter on monitoring is a good grounding for what to alert on): watch how long a normal deploy takes to go ready, and set the thresholds so a slow start is not mistaken for a dead one. If you use a hand-built load balancer, the rise and fall settings there are the same idea.