Test Coverage in Go and What the Numbers Do Not Tell You

Coverage is the easiest test metric to collect and the easiest to misread. Go ships the tooling in the standard distribution: no plugin, no instrumentation step to configure. A percentage appears next to every package in go test output once you ask for it. What it measures is narrower than most dashboards imply, and the gap between "this line ran" and "this line is checked" is where the interesting bugs live.

This article shows the commands, then spends most of its length on that gap, with a test that scores 100% and fails to notice a bug.

Collecting coverage

go test -cover ./...

That prints a statement percentage per package. For more detail, write a profile and read it:

go test -coverprofile=c.out ./...
go tool cover -func=c.out
go tool cover -html=c.out

-func prints one line per function and a total. -html opens a browser view of each file with covered statements in green and uncovered in red, which is the most useful form for finding what a test missed. -covermode picks how counts are kept: set records whether a statement ran, count records how many times, and atomic is the safe version for tests that run goroutines in parallel. go test -race already implies atomic. -coverpkg=./... attributes coverage to packages other than the one under test, which matters when an integration test in one package exercises code in several. Covering a compiled binary is a different mechanism, described in the binary coverage article.

A test that scores 100 percent

package grade

// Grade maps a score to a letter.
func Grade(score int) string {
	switch {
	case score >= 90:
		return "A"
	case score >= 80:
		return "B"
	case score >= 70:
		return "C"
	}
	return "F"
}

// Discount returns the percentage off for an order.
func Discount(total int, member bool) int {
	d := 0
	if total > 100 {
		d = 10
	}
	if member {
		d += 5
	}
	return d
}

package grade

import "testing"

// This test executes every line and checks almost nothing.
func TestEverythingRuns(t *testing.T) {
	for _, s := range []int{95, 85, 75, 10} {
		_ = Grade(s)
	}
	_ = Discount(150, true)
	_ = Discount(50, false)
}

func TestGradeBoundaries(t *testing.T) {
	cases := map[int]string{90: "A", 89: "B", 80: "B", 79: "C", 70: "C", 69: "F"}
	for score, want := range cases {
		if got := Grade(score); got != want {
			t.Errorf("Grade(%d) = %q, want %q", score, got, want)
		}
	}
}

TestEverythingRuns calls Grade with one score for each branch and calls Discount twice. It never compares a result with anything. That is a bad test, and the coverage tool has no way to tell.

echo "# weak test only"
go test -count=1 -run EverythingRuns -coverprofile=c.out . | sed -E 's/\t[0-9.]+s//'
echo
echo "# now break the code: >= 90 becomes > 90"
cp grade.go grade.go.orig
perl -pi -e 's/score >= 90/score > 90/' grade.go
go test -count=1 -run EverythingRuns -coverprofile=c.out . | sed -E 's/\t[0-9.]+s//'
echo
echo "# the boundary test notices"
go test -count=1 -run GradeBoundaries . 2>&1 | grep -E 'Grade|FAIL|ok' | sed -E 's/\t[0-9.]+s//' | sort | head -4
mv grade.go.orig grade.go
echo
echo "# per function view of the real code"
go test -count=1 -coverprofile=c.out . >/dev/null
go tool cover -func=c.out

# weak test only
ok  	example.test/lab	coverage: 100.0% of statements

# now break the code: >= 90 becomes > 90
ok  	example.test/lab	coverage: 100.0% of statements

# the boundary test notices
    grade_test.go:18: Grade(90) = "B", want "A"
--- FAIL: TestGradeBoundaries (0.00s)
FAIL
FAIL

# per function view of the real code
example.test/lab/grade.go:4:	Grade		100.0%
example.test/lab/grade.go:17:	Discount	100.0%
total:				(statements)	100.0%

The first block reports coverage: 100.0% of statements for the weak test. The second block edits the source: score >= 90 becomes score > 90, so a student with exactly 90 now gets a B. The weak test passes again, still at 100%. Nothing in it looked at the answer, so a wrong answer is as good as a right one.

The third block shows the test that would have caught it. TestGradeBoundaries checks the values on both sides of every boundary, and it fails with Grade(90) = "B", want "A". Its coverage number is no higher than the weak test's; its value is the assertions. Two tests with identical coverage can differ completely in strength.

What the percentage can and cannot say

Coverage answers one question: which statements ran. That makes it a good detector of one failure, code with no test at all, and a poor detector of everything else.

It cannot see assertions. As above, a test that executes code without checking it counts in full. Teams that gate on a number get tests written to move the number.

It counts statements, not conditions. if total > 100 && member is one statement. A test where the condition is false covers it as well as a test where it is true. Go has no branch or condition coverage, so Discount can reach 100% without ever testing total > 100 and member independently. Compare the table: the weak test hit both if bodies by luck of its inputs.

It cannot see missing code. If the function should have rejected a negative score and does not, there is no statement for coverage to flag. Coverage measures what was written, not what was required.

It cannot see interaction. Each statement may run in some test without ever running with the state that triggers the bug. Property tests and fuzzing explore combinations that line coverage does not.

What it is good for

• Finding forgotten code. The HTML view is a quick way to see that the error path of Open was never exercised. That is a prompt to write a test, and the best use of the tool.
• Reviewing a change. Coverage of the lines a pull request touched is more informative than the whole package's percentage. Many teams comment per file coverage on the diff.
• Tracking a trend. A package that falls from 82% to 60% over a quarter is telling you something even if neither number is a target.
• Finding dead code. A function with 0% in a thorough suite might be unused.

Setting a threshold without damage

If CI must have a number, make it a floor and not a goal. A project-wide minimum such as 70% stops the worst neglect and invites no gaming, while a requirement of 100% produces tests like the first one here. Better, ratchet it: fail when coverage drops by more than a point from the main branch. A ratchet punishes removing tests and ignores the absolute level, which varies sensibly across packages. Exclude generated code and main packages from the figure, either with build tags or by filtering the profile, because they inflate or deflate it for reasons that have nothing to do with test quality.

Pair the number with something that tests the tests. Mutation testing changes the code, as the lab did by hand, and checks that some test fails; the mutation testing article covers the tools and what they find. The testing guide puts coverage in context with the rest of the practice.

A habit that works better than a target

When you fix a bug, write the test first, watch it fail, then fix, and look at the coverage of the lines you touched. When you write a function with several branches, list the inputs that distinguish them before you open the coverage report, and compare. The tool is most honest when it confirms a list you made on purpose. It is least honest when you read it first and write tests until the colours change, because the tests then follow the report instead of the specification.

Review helps in the same way. In a code review, ask what each new test would do if the code under it were wrong. A test that still passes after you delete the body of the function it claims to cover is a test of nothing, and reviewers can see that without any tool.

Limits of this lab

The mutation here was applied by perl and was one edit. A real tool would try hundreds. The coverage figures are statement coverage in set mode from a single package, and they differ from the numbers a larger project reports because of -coverpkg, build tags and generated files. The tooling is documented in the Go blog's coverage post and the cmd/cover reference.