Service Level Objectives and Error Budgets for a Go API

Most teams run an API with a vague goal: it should be up and fast. "Up" and "fast" are not numbers. Without numbers, every conversation about reliability is an argument about feelings, the on-call engineer gets paged for noise or not paged for a real problem, and nobody can say whether it is safe to ship the risky change on Friday.

Service level objectives replace the feelings with arithmetic. This article defines the terms, works through the arithmetic with a simulated month of traffic written in Go, and turns it into alerts that catch both a sudden outage and a slow leak. The instrumentation side connects to tracing and the logging work; the alert design comes from the Google SRE workbook. It ran on Go 1.25.6.

The vocabulary

A service level indicator (SLI) is a measurement of how the service behaves, expressed as a ratio of good events to total events. "The proportion of requests that return a non-5xx response within 300 ms" is an SLI. Ratios are useful because they are always between 0 and 1, they aggregate by adding counts, and they behave under any traffic level.

A service level objective (SLO) is a target for the SLI over a time window: "99.9 percent of requests succeed over a rolling 30 days". It is an internal goal that engineering and the product agree on.

A service level agreement (SLA) is a contract with consequences, usually money, and should be looser than the SLO so you notice trouble before you owe anyone a refund.

The error budget is whatever the SLO does not require: 100 percent minus the target. At 99.9 percent it is 0.1 percent of requests, which you are allowed to fail. The budget turns reliability into something you can spend: deploys, experiments and incidents all draw on it. A service that is far inside its budget can ship faster. A service that has burned it should slow down and fix reliability first.

Choosing SLIs for an API

Pick indicators users would notice.

• Availability: the fraction of valid requests that get a successful response. Count 5xx and timeouts as bad. Do not count 4xx caused by clients (400, 401, 404) as bad, because a client mistake is the service working. Count 429 only if your own rate limiter fired on legitimate traffic.
• Latency: the fraction of requests that complete under a threshold, such as 300 ms. Prefer this to a percentile, because "99 percent under 300 ms" expressed as good over total events can be summed across instances and windows, and a percentile cannot.
• Correctness: the fraction of responses that are right, where you can check it (a reconciliation job comparing results).
• Freshness, for pipelines: the fraction of data updated within a bound.

Measure at the edge where the user's request arrives (the load balancer or the gateway), not inside a single instance, so an instance that dies and serves nothing is counted. Start with one or two SLIs per important user task. A handful of indicators that people read beat thirty that nobody does.

The arithmetic, with a month of simulated traffic

Reasoning about budgets is easier with numbers in front of you. The lab simulates 30 days of traffic at 1,000 requests a minute: a background error rate of 0.02 percent, a 45 minute outage on day 12 at 8 percent errors, and a six hour leak on day 20 at 0.8 percent. It computes the SLI, the budget and the burn rate with prefix sums over per-minute counters, which is what a metrics system stores as counters.

package main

import (
	"fmt"
	"math/rand"
	"time"
)

const (
	minutesPerDay = 24 * 60
	days          = 30
	reqPerMinute  = 1000
	sloTarget     = 0.999 // 99.9% of requests succeed over 30 days
)

// bucket is one minute of traffic, which is what a metrics system would keep as counters.
type bucket struct{ total, bad int }

func main() {
	rng := rand.New(rand.NewSource(42))
	start := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC)
	buckets := make([]bucket, days*minutesPerDay)

	for m := range buckets {
		rate := 0.0002 // background noise: 0.02% of requests fail
		day, minute := m/minutesPerDay, m%minutesPerDay
		switch {
		case day == 11 && minute >= 10*60 && minute < 10*60+45:
			rate = 0.08 // an outage: 45 minutes at 8% errors
		case day == 19 && minute >= 2*60 && minute < 8*60:
			rate = 0.008 // a slow leak: 6 hours at 0.8%, eight times the sustainable rate
		}
		n := reqPerMinute
		bad := 0
		for i := 0; i < n; i++ {
			if rng.Float64() < rate {
				bad++
			}
		}
		buckets[m] = bucket{n, bad}
	}

	// Prefix sums make "errors in the last W minutes" a subtraction.
	totalSum, badSum := make([]int, len(buckets)+1), make([]int, len(buckets)+1)
	for i, b := range buckets {
		totalSum[i+1] = totalSum[i] + b.total
		badSum[i+1] = badSum[i] + b.bad
	}
	window := func(end, w int) (total, bad int) {
		from := max(0, end+1-w)
		return totalSum[end+1] - totalSum[from], badSum[end+1] - badSum[from]
	}
	burn := func(end, w int) float64 { // how many times faster than "exactly on budget" the budget is being spent
		t, b := window(end, w)
		return float64(b) / float64(t) / (1 - sloTarget)
	}

	total, bad := window(len(buckets)-1, len(buckets))
	budget := float64(total) * (1 - sloTarget)
	fmt.Printf("30 days: %d requests, %d failed, SLI %.4f%%\n", total, bad, 100*(1-float64(bad)/float64(total)))
	fmt.Printf("error budget at 99.9%%: %.0f failed requests (about %.1f minutes of total outage)\n", budget, budget/reqPerMinute)
	fmt.Printf("budget used: %.0f%%, remaining: %.0f%%\n", 100*float64(bad)/budget, 100-100*float64(bad)/budget)

	// How much of the budget did each event cost? Count the extra failures above the background rate.
	cost := func(from, to int) float64 {
		_, b := window(to, to-from+1)
		return 100 * (float64(b) - float64(to-from+1)*reqPerMinute*0.0002) / budget
	}
	fmt.Printf("outage on day 12 used about %.0f%% of the budget\n", cost(11*minutesPerDay+600, 11*minutesPerDay+644))
	fmt.Printf("slow leak on day 20 used about %.0f%% of the budget\n", cost(19*minutesPerDay+120, 19*minutesPerDay+479))

	// Multi-window burn rate alerts, as in the Google SRE Workbook: both the long and the short window must exceed the threshold.
	type rule struct {
		name        string
		long, short int
		threshold   float64
	}
	rules := []rule{{"page: fast burn (1h and 5m above 14.4x)", 60, 5, 14.4}, {"ticket: slow burn (6h and 30m above 6x)", 360, 30, 6}}
	for _, r := range rules {
		fmt.Println(r.name)
		firing, from := false, 0
		stamp := func(m int) string {
			return fmt.Sprintf("day %d %s", m/minutesPerDay+1, start.Add(time.Duration(m)*time.Minute).Format("15:04"))
		}
		for m := range buckets {
			on := m >= r.long && burn(m, r.long) > r.threshold && burn(m, r.short) > r.threshold
			if on && !firing {
				firing, from = true, m
				fmt.Printf("    fired    %s\n", stamp(m))
			}
			if !on && firing {
				firing = false
				fmt.Printf("    resolved %s (after %d minutes)\n", stamp(m), m-from)
			}
		}
	}

	// What a typical "error rate above 2% for the last 5 minutes" alert does with the same data.
	naiveOutage, naiveLeak := 0, 0
	for m := 5; m < len(buckets); m++ {
		if burn(m, 5) > 20 { // 2% is 20 times the 0.1% budget rate
			if m/minutesPerDay == 11 {
				naiveOutage++
			}
			if m/minutesPerDay == 19 {
				naiveLeak++
			}
		}
	}
	fmt.Printf("naive alert (5m error rate above 2%%): true for %d minutes on day 12, %d minutes on day 20\n", naiveOutage, naiveLeak)
}

30 days: 43200000 requests, 15056 failed, SLI 99.9651%
error budget at 99.9%: 43200 failed requests (about 43.2 minutes of total outage)
budget used: 35%, remaining: 65%
outage on day 12 used about 8% of the budget
slow leak on day 20 used about 7% of the budget
page: fast burn (1h and 5m above 14.4x)
    fired    day 12 10:10
    resolved day 12 10:49 (after 39 minutes)
ticket: slow burn (6h and 30m above 6x)
    fired    day 12 10:26
    resolved day 12 11:12 (after 46 minutes)
    fired    day 20 06:32
    resolved day 20 08:07 (after 95 minutes)
naive alert (5m error rate above 2%): true for 47 minutes on day 12, 0 minutes on day 20

The totals. 43.2 million requests, about 15,000 failed, an SLI of about 99.965 percent. Against a 99.9 percent target the budget is 43,200 failed requests. Divide that by the 1,000 requests a minute and you get 43.2 minutes: the same budget expressed as "how long can we be completely down". That conversion is worth keeping in your head: 99.9 percent is about 43 minutes a month, 99.99 percent about 4.3 minutes, 99 percent about 7.2 hours.

What each event cost. The outage was dramatic and used about 8 percent of the budget. The leak, unremarkable on a graph, used about 7 percent. Both together plus background noise used 35 percent, leaving 65. The point is not that one is worse; it is that the budget measures consequences in the same unit for every kind of failure, short or long, loud or quiet.

Burn rate

The burn rate is how fast you are spending relative to the pace that would exactly exhaust the budget at the end of the window. A burn rate of 1 means you will finish the month at exactly 100 percent used. A burn rate of 10 would empty the budget in three days. Written as a formula: error ratio over some window divided by the allowed error ratio (here 0.001).

Burn rate is what to alert on, for two reasons. It is independent of traffic level, and it speaks about the budget, which is what you care about: "at this rate we run out in 2 days" tells you how urgent a problem is, and a raw error percentage does not.

Alerting on burn rate

A single threshold on a single window has a bad trade-off. A short window with a high threshold reacts quickly and misses slow burns. A long window with a low threshold catches slow burns and takes hours to notice an outage, and keeps alerting long after it ended. The fix, from the Google SRE workbook, is to require two windows at once: a long one to establish that the problem is real and a short one to confirm it is still happening.

The lab uses two rules:

A burn rate of 14.4 sustained for an hour spends 2 percent of a 30 day budget. A burn rate of 6 sustained for six hours spends 5 percent. Those are the workbook's suggested thresholds, and they are sensible starting points.

What the run shows. The fast-burn rule fired at 10:10 on day 12, ten minutes after the outage started. It needed ten minutes because the one hour window had to accumulate 1.44 percent errors, and at 8 percent that takes a little under 11 minutes. It resolved at 10:49, four minutes after the outage ended: the short window clearing is what releases the alert, which is the second benefit of the pair. A single one hour window would have kept firing for an hour after recovery.

The slow-burn rule also fired during the outage, at 10:26, and resolved at 11:12. On day 20 it fired again at 06:32, four and a half hours into the leak, once the six hour window held enough bad requests to exceed a burn rate of 6. The fast rule never fired for the leak, correctly: at 0.8 percent errors it is eight times the sustainable rate, which is a ticket, not a 3 a.m. phone call.

Compare the last line of the output: a typical alert on "error rate above 2 percent over 5 minutes" fired for 47 minutes on day 12 and for 0 minutes on day 20. It saw the outage and missed a failure that consumed seven percent of the month's budget. Leaks like that are common in practice: a bad dependency, a slow memory leak, a malformed client. A budget-based alert is what turns them into tickets.

Wiring it up with Prometheus

In production, the counters come from instrumentation, and a monitoring system does the windowed arithmetic. The usual setup in Go is the Prometheus client library with a counter by status class, plus a histogram for latency, recorded in a middleware:

var requests = promauto.NewCounterVec(prometheus.CounterOpts{
	Name: "http_requests_total",
	Help: "Requests by route template and status class.",
}, []string{"route", "class"})

func instrument(route string, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		rec := &statusRecorder{ResponseWriter: w, status: 200}
		next.ServeHTTP(rec, r)
		requests.WithLabelValues(route, strconv.Itoa(rec.status/100)+"xx").Inc()
	})
}

Label by the route template (/orders/{id}), never the raw path, or the label set grows without bound. Then a recording rule computes the error ratio over each window, and the alert combines two of them:

- alert: ErrorBudgetFastBurn
  expr: |
    ( sum(rate(http_requests_total{class="5xx"}[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001) )
    and
    ( sum(rate(http_requests_total{class="5xx"}[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001) )
  labels: { severity: page }

I did not run Prometheus for this article; the expressions follow its documented rate semantics and the workbook's alerting recipe. Test your own rules by replaying a recorded outage through promtool test rules. The Prometheus documentation covers alert design in more detail.

A latency SLI that adds up

Latency is the SLI teams most often get wrong, by alerting on a p99 computed per instance. Percentiles cannot be averaged or summed, so the number depends on how many instances you have and how traffic is spread among them. Define latency as a ratio of events instead: requests faster than 300 ms divided by all requests. With a Prometheus histogram that has a bucket boundary at 0.3 seconds, the numerator is the le="0.3" bucket counter and the denominator is the _count series, both of which add across instances and windows. The same burn-rate arithmetic then works unchanged.

Pick the threshold from what the user experiences, and make sure the histogram has a bucket exactly at it, because you can only read the proportion at a boundary. If the SLO is 99 percent of requests under 300 ms, add a boundary at 0.3 and look at the neighbouring buckets at 0.25 and 0.5, which tell you whether you are close to the edge. It is common to define two latency objectives, say 90 percent under 100 ms and 99 percent under 500 ms, so that both the typical and the tail experience are covered.

When traffic is low

Burn rates assume enough events per window that a single failure does not move the ratio. A service with ten requests an hour has a 5 minute window with zero or one request, and one failed request is a 100 percent error rate. Three approaches help. Use longer windows. Alert on the count of failures in addition to the ratio, so a single error does not page. Or generate synthetic traffic (a probe that exercises the main path every few seconds) so the service has a steady baseline to measure. A probe is also a good way to know that the service works when real users are asleep, though it only measures what it exercises.

For services with bursty traffic, such as batch endpoints, measure per job or per batch and not per minute.

Mistakes to avoid

Counting everything as a request. Health checks, bot traffic and your own load tests dilute the ratio and hide real problems. Exclude them in the query by label, or measure only on user routes.

Counting nothing as bad. If a failure returns 200 with an error in the body, or times out at the client without the server noticing, your server-side counters are blind to it. Measure at the load balancer, which sees the timeouts, and use the outcome the client saw.

Alerting on SLO violations after the fact. The budget is a forecast tool. An alert that fires when the month's SLI crosses 99.9 percent tells you after it is too late to act. Burn-rate alerts exist so you hear about it while there is still budget to protect.

One SLO for everything. A bulk export and a login call do not deserve the same target. Split by user task or by tier, and give the critical few strict objectives.

Choosing the target

Do not pick 99.99 percent because it sounds professional. Each extra nine costs roughly ten times more in redundancy, process and engineering time, and users may not notice the difference past the reliability of their own network. Reasonable starting points are 99.5 to 99.9 percent for an internal or early-stage API, 99.9 percent for a customer-facing one and higher only for things people cannot work without.

A good method: measure what you achieve today over the last quarter, set the SLO a little below it, and tighten it when the team can sustain more. An SLO you break every month is noise. An SLO you never come near is a target set too low to inform any decision.

Remember dependencies. Your availability is bounded by the product of your hard dependencies' availability: three components at 99.9 percent each give about 99.7 percent. If the database and the payment provider are in every request's path, your target must reflect that, or you need caches, fallbacks and timeouts that let you degrade (timeouts and retries).

The error budget policy

The numbers only change behaviour if the team agrees in advance what happens when the budget runs low. A short written policy is enough:

• Above 50 percent remaining: ship normally.
• Below 25 percent: risky changes need an extra review; prioritise reliability work in the next sprint.
• Budget exhausted: feature releases stop until the burn is understood and fixed, except for fixes.
• A single incident above 20 percent of the budget gets a post-mortem.

Agree it with product management before you need it. A policy invented in the middle of an incident is a negotiation, not a policy.

Reporting

Publish the SLO, the current SLI and the budget remaining where the whole team looks: a dashboard, a weekly message. Track budget consumed per cause, so the post-mortem list and the budget agree about what hurts most. When an SLO is missed, record why and what changed, because the pattern across quarters teaches more than any single incident.

Start small: pick one critical endpoint or user task, define a good/bad event, record it for a few weeks, and look at the numbers before choosing a target. The Google SRE book has chapters on service level objectives and the workbook on alerting on SLOs, which is the source of the multi-window recipe used above.