Load Test Go Services With k6 and Read the Results

A unit test tells you the code works for one caller. A load test tells you what it does for a thousand. The questions are different: how many requests per second before latency degrades, where the tail goes under load, what breaks first, and whether a change made things worse. Go services are often fast enough that these questions never come up until traffic does.

k6 is a popular tool for this: scenarios are JavaScript, the engine is written in Go, and results are summarised with percentiles and checks. This article shows a k6 script and how to read the output. I did not have k6 installed for this article, so the script below is described from the k6 documentation and not run. The reading-results half is run: a Go program with the same arithmetic produced the percentiles quoted.

A script with stages and thresholds

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 20 },
    { duration: '1m', target: 20 },
    { duration: '10s', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<200', 'p(99)<500'],
  },
};

export default function () {
  const res = http.get('http://localhost:8080/api/items');
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

options.stages describes the number of virtual users over time: ramp to 20 in 30 seconds, hold for a minute, ramp down in 10. A virtual user runs the default function in a loop, so each one makes a request, checks it and sleeps a second. At 20 users that is about 20 requests a second. To model arrival rate rather than user count, use the constant-arrival-rate executor, which keeps the request rate fixed even when the service slows down, and exposes queueing that a closed model hides.

thresholds turn the run into a test. If more than 1% of requests fail, or the 95th percentile is over 200 ms, or the 99th over 500 ms, k6 exits non-zero and a CI job fails. Choose thresholds from your service's objectives, not from the first run's output.

Run it with k6 run script.js. For anything you will quote as a number, run k6 on a different machine from the service, since the two compete for CPU otherwise.

Why percentiles, not the average

Here is the same arithmetic in Go. A handler answers in 2 ms, except for 5% of requests that take 40 ms, standing for a cache miss or a slow query. A load generator sends 2000 requests from 20 concurrent workers and prints the distribution.

package main

import (
	"fmt"
	"math/rand/v2"
	"net/http"
	"net/http/httptest"
	"sort"
	"sync"
	"time"
)

// A service with a fast path and a slow tail, standing in for your handler.
func handler() http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		d := 2 * time.Millisecond
		if rand.IntN(100) < 5 {
			d = 40 * time.Millisecond
		}
		time.Sleep(d)
		fmt.Fprint(w, "ok")
	})
}

func main() {
	srv := httptest.NewServer(handler())
	defer srv.Close()

	const workers, perWorker = 20, 100
	var mu sync.Mutex
	var lat []time.Duration
	var wg sync.WaitGroup
	start := time.Now()
	for range workers {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for range perWorker {
				t0 := time.Now()
				resp, err := http.Get(srv.URL)
				if err != nil {
					continue
				}
				resp.Body.Close()
				mu.Lock()
				lat = append(lat, time.Since(t0))
				mu.Unlock()
			}
		}()
	}
	wg.Wait()
	elapsed := time.Since(start)
	sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
	p := func(q float64) time.Duration { return lat[int(float64(len(lat)-1)*q)].Round(time.Millisecond) }
	fmt.Printf("requests=%d workers=%d\n", len(lat), workers)
	fmt.Printf("p50=%v p90=%v p95=%v p99=%v max=%v\n", p(0.50), p(0.90), p(0.95), p(0.99), lat[len(lat)-1].Round(time.Millisecond))
	var sum time.Duration
	for _, d := range lat {
		sum += d
	}
	fmt.Printf("mean=%v throughput_above_1000_rps=%v\n", (sum / time.Duration(len(lat))).Round(100*time.Microsecond), float64(len(lat))/elapsed.Seconds() > 1000)
}

requests=2000 workers=20
p50=3ms p90=4ms p95=41ms p99=42ms max=46ms
mean=5.6ms throughput_above_1000_rps=true

The result is a distribution with two humps. The median is 3 ms and the 90th percentile 4 ms, so 90% of users see a fast service. The 95th percentile is 41 ms, and the 99th is 42 ms. The maximum is 46 ms. The mean, 5.6 ms, falls between the two humps and describes nobody: no request took 5.6 ms.

A threshold on the average would have passed. A threshold on p(95)<200 passes too in this case (41 ms is under it), but a threshold at p(95)<20 would fail, and that is the kind of statement users feel. When the tail is more than ten times the median, the slowest 5% of requests are what a person refreshing the page notices. For services that fan out to several backends, the tail matters more still, because a page that makes ten calls hits the slow path in about 40% of loads at a 5% slow rate per call.

Also read the maximum, with suspicion. One slow request in 2000 sets the max and tells you little; the p99 and p99.9 are steadier. Report the sample size with every percentile, because p99 of 200 requests is the 2nd slowest.

Reading a k6 summary

The end-of-test summary reports, for each metric, the average, minimum, median, maximum and percentiles. Focus on four lines.

• http_req_duration: the full request time. p(95) and p(99) are the ones to watch. k6 also splits it into http_req_waiting (time to first byte, which is the server's work) and http_req_connecting, http_req_sending and http_req_receiving, so you can tell the server from the network.
• http_req_failed: the fraction of failed requests. Anything above zero under steady load deserves an explanation.
• http_reqs: the total count and the rate per second. This is the throughput the service sustained.
• checks: the pass rate of your assertions. A fast service that returns errors passes the duration threshold and fails the checks, so always check the status and a piece of the body.

Compare runs, not single numbers. Run the same script before and after a change, from the same machine, and look at the p95 and the failure rate. For repeated comparisons the discipline in the benchstat article applies: variation between runs is real, and a 5% difference means nothing without repeats.

What to look for when the numbers are bad

• Latency climbs steadily as load rises, with no errors. The service is saturating a resource: CPU, a connection pool, a lock. Profile under load with pprof.
• Latency is flat and then errors appear. A limit was hit: file descriptors, MaxConns, a rate limiter, an upstream. The first error's message names it.
• Latency is bimodal, as in the lab: a slow path exists. Find what differs between fast and slow requests with tracing or a label on the metric.
• Throughput plateaus below the target while latency is fine. The load generator is the bottleneck. Check its CPU.
• The results degrade over a long run. A leak or a growing structure. Keep an eye on memory and goroutine counts during a soak test.

Go services make some of these easy to see. net/http/pprof exposes CPU and goroutine profiles, runtime/metrics gives GC pause and heap figures, and a Prometheus histogram on the handler gives the server's own view to compare with the client's. A load test with no server-side metrics tells you that it is slow and not why.

Where load tests mislead

A load test is a model, and models fail. The traffic is synthetic: real users do not all request the same URL with the same headers, and a cache that serves your one URL from memory makes the service look faster than it is. The environment is smaller than production, so absolute numbers do not transfer, and the ratios between changes only partly do. The data is small: a database with 1,000 rows behaves differently from one with 100 million. And closed-model tools like the script above slow down their own request rate when the service slows down, which understates how bad overload is.

Mitigate by replaying a sample of production traffic shapes, seeding realistic data volumes, using arrival-rate executors for overload tests, and treating the first numbers as a baseline for comparison, not a capacity promise. Run load tests in a pipeline before releases for the services where latency is part of the contract, and run a small smoke version on every merge so a catastrophic regression fails fast.

The k6 documentation explains the executors and thresholds, and the Go side of profiling is in the net/http/pprof package. For the lower-level view, see benchmarks, and for the API level, the load testing note in the APIs cluster.