Database Health Checks and Startup Retries in Go Services

A service and its database do not always start in the right order. A deploy restarts both, a node reboots, a managed database fails over, a Docker Compose stack brings everything up at once. A Go service that opens the database and exits on the first error will crash loop until the database happens to be ready, and one that never checks will accept traffic it cannot serve. Two small pieces of code fix both: a start-up wait with backoff, and health endpoints that say the right thing about the database. This article builds them on Go 1.25.6 with pgx and PostgreSQL 18.6, starts a database late on purpose, then freezes it, and records what the probes answered.

It belongs to the database cornerstone. The pool behind the ping is described in how database/sql works, and the migration step that often runs at the same point is covered in embedding migrations.

Why sql.Open is not enough

sql.Open does not connect. It validates its arguments and returns a pool, so a wrong password or a database that is down shows up on the first query, in the first request. db.PingContext forces a connection, which is why a start-up routine should call it. The question is what to do when it fails.

Exiting is a legitimate answer when the platform restarts you: Kubernetes will restart a crashed container with its own backoff. It is also noisy, it hides the cause in restart counts and it makes every deploy of a dependent system a flurry of failures. Waiting inside the process, bounded and logged, is gentler.

The lab

package main

import (
	"context"
	"database/sql"
	"errors"
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
	"os/exec"
	"strings"
	"time"

	_ "github.com/jackc/pgx/v5/stdlib"
)

const dsn = "postgres://postgres:[email protected]:55630/lab?sslmode=disable&connect_timeout=1"

// waitForDB retries a ping with exponential backoff until it works or the context ends.
func waitForDB(ctx context.Context, db *sql.DB) error {
	delay := 200 * time.Millisecond
	for attempt := 1; ; attempt++ {
		pctx, cancel := context.WithTimeout(ctx, time.Second)
		start := time.Now()
		err := db.PingContext(pctx)
		cancel()
		if err == nil {
			fmt.Printf("attempt %2d: connected\n", attempt)
			return nil
		}
		msg := err.Error()
		if i := strings.LastIndex(msg, ": "); i >= 0 {
			msg = msg[i+2:]
		}
		fmt.Printf("attempt %2d: %-34s after %-6v waiting %v\n", attempt, msg, time.Since(start).Round(10*time.Millisecond), delay)
		select {
		case <-ctx.Done():
			return fmt.Errorf("gave up: %w", errors.Join(ctx.Err(), err))
		case <-time.After(delay):
		}
		if delay *= 2; delay > 2*time.Second {
			delay = 2 * time.Second
		}
	}
}

// health has a liveness probe that never touches the database and a readiness probe that does.
func health(db *sql.DB) http.Handler {
	mux := http.NewServeMux()
	mux.HandleFunc("/livez", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "ok") })
	mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
		ctx, cancel := context.WithTimeout(r.Context(), 500*time.Millisecond)
		defer cancel()
		if err := db.PingContext(ctx); err != nil {
			http.Error(w, "database unavailable", http.StatusServiceUnavailable)
			return
		}
		fmt.Fprintln(w, "ready")
	})
	return mux
}

func main() {
	db, err := sql.Open("pgx", dsn)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	ctx, cancel := context.WithTimeout(context.Background(), 40*time.Second)
	defer cancel()
	start := time.Now()
	if err := waitForDB(ctx, db); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("database was ready after about %v\n", time.Since(start).Round(time.Second))

	srv := httptest.NewServer(health(db))
	defer srv.Close()
	probe := func(label, path string) {
		start := time.Now()
		resp, err := http.Get(srv.URL + path)
		if err != nil {
			log.Fatal(err)
		}
		resp.Body.Close()
		fmt.Printf("%-34s %s -> %d in %v\n", label, path, resp.StatusCode, time.Since(start).Round(50*time.Millisecond))
	}
	probe("database up", "/readyz")

	// The lab freezes the database container to stand in for an outage.
	_ = exec.Command("docker", "pause", "gh-lab-pg630").Run()
	probe("database frozen", "/readyz")
	probe("database frozen", "/livez")
	_ = exec.Command("docker", "unpause", "gh-lab-pg630").Run()
	time.Sleep(300 * time.Millisecond)
	probe("database back", "/readyz")
}

docker rm -f gh-lab-pg630 >/dev/null 2>&1
(sleep 4; docker run -d --name gh-lab-pg630 -p 127.0.0.1:55630:5432 -e POSTGRES_PASSWORD=secret -e POSTGRES_DB=lab postgres:18-alpine >/dev/null) &
go run .
docker rm -f gh-lab-pg630 >/dev/null 2>&1

attempt  1: connection refused                 after 0s     waiting 200ms
attempt  2: connection refused                 after 0s     waiting 400ms
attempt  3: connection refused                 after 0s     waiting 800ms
attempt  4: connection refused                 after 0s     waiting 1.6s
attempt  5: connection refused                 after 0s     waiting 2s
attempt  6: connected
database was ready after about 5s
database up                        /readyz -> 200 in 0s
database frozen                    /readyz -> 503 in 500ms
database frozen                    /livez -> 200 in 0s
database back                      /readyz -> 200 in 0s

The script removes any earlier lab container, starts a background job that launches a fresh postgres:18-alpine container after four seconds, and then runs the program. So the service starts against a port where nothing listens, which is the situation after a simultaneous deploy. The program's output follows in three parts.

The start-up wait

waitForDB pings with a one second timeout per attempt, prints the failure, sleeps, doubles the sleep up to a cap of two seconds, and tries again until the overall context ends. The output:

• attempts 1 to 5 failed with connection refused, each instantly, and the waits between them were 200ms, 400ms, 800ms, 1.6s and 2s, the last at the cap;
• attempt 6 connected, and the program reported the database ready after about 5 seconds.

Four things in that function are choices.

A timeout on every attempt. Each ping runs with context.WithTimeout(ctx, time.Second). Refused connections fail at once, but a database that accepts the connection and then stalls, or an address that drops packets, would otherwise block each attempt for as long as the operating system allows, which can be minutes. The connect_timeout=1 in the connection string bounds the dial the same way.

A cap on the backoff. Doubling forever would reach minutes. Two seconds keeps the service responsive once the database is up.

A total deadline. The outer context gives the whole wait 40 seconds. After that the function returns an error that joins the context error and the last ping error, and the program exits with a clear message. A service that waits forever is a service nobody notices is not working. Choose the deadline above the database's usual start time and below your platform's patience.

Jitter belongs in production. The lab's delays are exact so the output is readable. In a deployment with fifty replicas that all start together, identical delays make all fifty retry in the same instant, which can flatten a database that is trying to come up. Add a random amount, for example a uniform value between half and the full delay, to each wait.

What counts as retryable matters. Connection refused, a timeout and "the database system is starting up" (SQLSTATE 57P03) are worth retrying. An authentication failure (28P01) or a missing database (3D000) will not fix itself in five seconds, and retrying hides a configuration error behind a delay; fail fast on those, using the error handling in database errors.

Liveness and readiness are different questions

Container platforms ask a service two things. Liveness: is this process stuck, should I restart it? Readiness: should I send it traffic right now? They need different answers about the database.

The lab's health handler has one probe for each:

• /livez never touches the database. It returns 200 if the process can answer. If it pinged the database, a database outage would make every liveness probe fail, and the platform would restart every pod at once. The restarts fix nothing, they drop connections and caches, and when the database returns the service comes back to a cold start all together. Liveness should detect deadlocks inside the process, not problems elsewhere.
• /readyz pings the database with a short timeout of 500ms and answers 503 on failure. A not ready pod is taken out of the load balancer and left running, and it is put back when the ping succeeds again. That is the correct reaction to a database outage: stop sending requests that would fail, and recover when it is back.

The second half of the output shows it. With the database up, /readyz returned 200 in 0s. The lab then froze the database container with docker pause, which keeps the port open and stops the process, the nastiest kind of outage because connections hang instead of failing. /readyz returned 503 after 500ms, exactly the probe's timeout, and /livez still returned 200 in 0s. After docker unpause, /readyz returned 200 again. Without the timeout on the ping, the readiness request would have hung until the platform's probe timeout, and the result would have been reported the same way but with a stuck handler behind it.

Making the readiness check good

• Keep it cheap. PingContext borrows a connection and sends a trivial round trip. Do not run a real query, and do not run it on every request: the platform probes every few seconds, and that is enough.
• Do not let the probe exhaust the pool. With a tiny pool, a probe that waits for a free connection competes with real traffic. The 500ms timeout bounds it, and a probe that fails because the pool is busy is, accurately, a sign of trouble.
• Cache the result for a second or two if many things call it. A burst of probes should not become a burst of pings.
• Include only dependencies the service cannot work without. If the cache is optional, a cache outage should not make the pod unready. A readiness probe that checks everything turns one failure into a total outage.
• Report detail separately. A public 503 with no body is right for probes. A diagnostics endpoint, kept behind authentication, can say which dependency failed and for how long, and can export pool counters from db.Stats().
• Decide what a failover looks like. After a failover, connections to the old primary fail on their next use. The pool discards a bad connection and opens another, and SetConnMaxLifetime bounds how long a stale one can live. A readiness probe goes red for the seconds that takes, and returns to green by itself.

Start-up order inside the service

A tidy start-up has an order: read configuration, wait for the database, run migrations if the service owns them, warm what must be warm, then start listening. Starting the HTTP listener before the database is ready, with readiness returning 503, is also fine and lets the platform see the process as alive from the start. Doing it the other way, listening and serving real requests before the pool works, fails requests that a short wait would have saved.

Avoid two mistakes: calling log.Fatal inside the retry loop for a retryable error, and retrying inside request handlers as a way of hiding an outage. The first crash loops, and the second ties up goroutines and connections while clients wait; fail the request quickly and let readiness take the pod out of rotation.

Docker Compose

Compose has its own answer, depends_on with a condition: service_healthy and a healthcheck using pg_isready. It orders the start of containers on one machine and does nothing for a restart of the database later, a failover or Kubernetes. Use it as a convenience for development, and keep the retry in the service as the real mechanism, since it works everywhere. The probes' semantics are described in the Kubernetes documentation on liveness, readiness and startup probes, and the readiness side of a database check is a single PingContext, documented in the database/sql reference.