Load Test a Go API With k6 and vegeta and Read the Results

A load test answers one question: how does this service behave when traffic is high, and where does it stop coping? Getting a number is easy. Getting a number that means something is the hard part, and most bad load tests fail in the same way: they measure how fast the client can go, not how the server behaves under a load that arrives regardless of how it is doing.

This article uses two tools. vegeta is a Go library and command that sends requests at a constant rate. k6 is a JavaScript-scripted load tester, popular for CI and scenarios. The example runs vegeta as a library inside a Go program so the whole experiment is one file, and shows a k6 script for the same job. It ran on Go 1.25.6. For the profile-and-fix side after a load test finds a problem, see benchmarking an API handler and profiling with pprof.

Two kinds of load generator

A closed-loop generator runs a fixed number of workers. Each sends a request, waits for the answer, and sends the next. The load it produces depends on the server: when the server slows down, the workers send fewer requests per second. Apache Bench with a concurrency of 10 and wrk in its default mode behave this way.

An open-model generator sends requests at a fixed rate, whatever the server does. If the server slows down, requests pile up. That is what real traffic does: users do not wait for each other before clicking.

The difference matters most when the server is struggling. A closed-loop test politely backs off exactly when it should be applying pressure. This is the measurement problem known as coordinated omission, covered in depth in coordinated omission in load tests: the slow responses are under-counted because the client stops asking while it waits. An open-model test avoids it by construction, because it keeps the schedule whether or not the previous answer arrived.

The experiment

The server in the lab has two slots and each request takes 20 ms, so its capacity is about 100 requests a second. A request that arrives when both slots are busy waits for one, which is a queue. The program runs a closed-loop test with two workers, then vegeta at 50 requests a second (half the capacity) and at 200 (double).

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"sort"
	"sync"
	"time"

	vegeta "github.com/tsenart/vegeta/v12/lib"
)

// slowAPI can serve two requests at once and each takes 20 ms, so its capacity is about 100 requests a second.
func slowAPI() *httptest.Server {
	slots := make(chan struct{}, 2)
	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		slots <- struct{}{} // wait for a free slot: this is the queue that builds under overload
		time.Sleep(20 * time.Millisecond)
		<-slots
		w.Write([]byte(`{"ok":true}`))
	}))
}

// closedLoop is how a naive benchmark works: N workers, each sends the next request only after the last one returned.
func closedLoop(url string, workers int, d time.Duration) (n int, mean time.Duration) {
	var mu sync.Mutex
	var total time.Duration
	var wg sync.WaitGroup
	stop := time.Now().Add(d)
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for time.Now().Before(stop) {
				start := time.Now()
				resp, err := http.Get(url)
				if err != nil {
					return
				}
				resp.Body.Close()
				mu.Lock()
				n++
				total += time.Since(start)
				mu.Unlock()
			}
		}()
	}
	wg.Wait()
	return n, total / time.Duration(n)
}

func attack(url string, rps int, d time.Duration) *vegeta.Metrics {
	targeter := vegeta.NewStaticTargeter(vegeta.Target{Method: "GET", URL: url})
	attacker := vegeta.NewAttacker(vegeta.Timeout(10 * time.Second))
	var m vegeta.Metrics
	for res := range attacker.Attack(targeter, vegeta.Rate{Freq: rps, Per: time.Second}, d, "demo") {
		m.Add(res)
	}
	m.Close()
	return &m
}

func main() {
	srv := slowAPI()
	defer srv.Close()

	n, mean := closedLoop(srv.URL, 2, 2*time.Second)
	fmt.Printf("closed loop, 2 workers:   %4d requests, mean latency %v\n", n, mean.Round(time.Millisecond))

	for _, rps := range []int{50, 200} {
		m := attack(srv.URL, rps, 3*time.Second)
		codes := make([]string, 0)
		for c, k := range m.StatusCodes {
			codes = append(codes, fmt.Sprintf("%s:%d", c, k))
		}
		sort.Strings(codes)
		fmt.Printf("open model at %3d req/s: %4d requests, success %.0f%%, p50 %v, p99 %v, max %v\n",
			rps, m.Requests, m.Success*100, m.Latencies.P50.Round(time.Millisecond), m.Latencies.P99.Round(10*time.Millisecond), m.Latencies.Max.Round(10*time.Millisecond))
	}
}

closed loop, 2 workers:    172 requests, mean latency 23ms
open model at  50 req/s:  150 requests, success 100%, p50 21ms, p99 30ms, max 30ms
open model at 200 req/s:  600 requests, success 100%, p50 1.646s, p99 3.43s, max 3.46s

Reading the results

The closed-loop run looked fine. Two workers sent 172 requests in two seconds, with a mean latency of 23 ms. Nothing in that line says the server is at its limit. It is, in the sense that the two workers have exactly saturated the two slots, and the test cannot send more than the server can finish.

At 50 requests a second, the server is comfortable. All 150 requests succeeded. The median is 21 ms, just the service time of 20 ms, and p99 is 30 ms. This is the picture of a healthy service.

At 200 requests a second, it is not. The test sent 600 requests over three seconds, one every 5 ms, to a server that completes one every 10 ms. The queue grew for the whole run. The median latency was 1.646 s, p99 was 3.43 s and the worst request waited 3.46 s. And success was 100 percent, because every request eventually got a 200.

This last point is the one to take away. A load test that only counted errors would have passed the overloaded run. Timeouts, error rates and latency percentiles are different signals, and a useful test states a threshold on each: "p99 under 300 ms and fewer than 0.1 percent errors at 150 requests a second".

Read percentiles, not averages

The mean hides the tail, and the tail is what users feel. If 1 percent of requests take two seconds and the rest take 20 ms, the mean is about 40 ms and looks fine. The p99 says two seconds.

Report p50 (typical), p95 and p99 (the tail), and max (the worst case, noisy). For a service that handles pages made of many calls, even p99 understates the user's experience, because a page that makes 20 calls hits the p99 of at least one of them about 18 percent of the time. When latencies matter, look at p99.9 and keep the raw histogram.

Latency distributions are rarely normal. Do not average percentiles across runs or machines; merge the histograms. vegeta's results can be written to a file and re-reported with vegeta report -type=hdrplot for the full distribution.

The same test in k6

k6 scripts are JavaScript, run by a Go binary. The equivalent of the open-model run is a constant-arrival-rate scenario, which is k6's name for it:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    steady: {
      executor: 'constant-arrival-rate',
      rate: 200, timeUnit: '1s', duration: '30s',
      preAllocatedVUs: 50, maxVUs: 500,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(95)<200', 'p(99)<400'],
  },
};

export default function () {
  const res = http.get('http://localhost:8080/api/items');
  check(res, { 'status is 200': (r) => r.status === 200 });
}

The thresholds block is what makes it a test and not a benchmark: k6 exits non-zero if the 95th percentile exceeds 200 ms or more than 0.1 percent of requests fail, so a CI job can gate on it. The default executor (constant-vus) is closed-loop, so if you copy a tutorial script without setting executor, you are doing the thing the first section warned about. I did not run k6 for this article; the script follows its documented scenarios, and the vegeta lab is the measured part.

Which tool to use is mostly taste. vegeta is a single Go dependency, composes in Go code, and is excellent for steady-rate tests and for piping results into reports. k6 has better scenario scripting (login then browse, ramp up and down, mixed endpoints), built-in thresholds and ecosystem integrations for dashboards. For multi-step user journeys, choose k6. For hammering one endpoint at a known rate, vegeta is quicker.

Designing a useful test

Know what you are testing. A load test of /health tells you about the router. Hit the endpoints that matter, with realistic data: the same database size, the same distribution of ids, the same request bodies. Hitting one cached row repeatedly tests the cache.

Warm up, then measure. Connection pools, JIT-free as Go is, still have caches, DNS, TLS sessions and the database buffer pool to warm. Discard the first 10 to 30 seconds.

Step the rate. Run at 50 percent, 75 percent, 100 percent and 150 percent of expected peak, and find the knee where latency rises steeply. That knee is the real capacity. The 50 and 200 comparison above is that idea in two points.

Check the generator. If the load generator is saturated, the numbers describe the generator. Watch its CPU. Use a separate machine or container for it. On a laptop, the client and server compete for the same cores, which inflates latency in both directions.

Watch the server, not just the client. During the run, record CPU, memory, goroutine count, GC pause time, open connections, database connection pool wait time and queue depths. The bottleneck is usually in one of those, and the client's view only tells you something is wrong.

Test realistic failure. Kill a replica mid-test. Slow down the database. Do the retries and timeouts from the previous article on timeouts and retries hold up, or do they amplify?

Keep the result. Write down the commit, the machine, the rate, the duration and the percentiles. Without a baseline, "it feels slower" is all you have next quarter.

Interpreting the knee

When latency rises steeply as the rate approaches capacity, the system behaves like a queue: utilisation near 100 percent means waiting time explodes. The lab's server shows the extreme case. Once the arrival rate exceeds the service rate, the queue grows without bound for as long as the load lasts, and every request that arrives later waits longer. Real services escape this in one of three ways: they shed load (answering 503 quickly), they apply backpressure by limiting concurrency, or they time out clients. Which one you have is something a load test should confirm. A service with no limit does not fail cleanly, it just gets slower for everyone until something upstream gives up.

That makes the test's second job: confirm the failure mode. Push past capacity and check that the service returns fast errors for the excess (load shedding), keeps serving the admitted requests at healthy latency, and recovers quickly when the load drops. The vegeta README documents the command line usage and library API used above.