Mutation Testing in Go and Whether It Finds Anything

A coverage report says which lines ran. A mutation report says whether the tests would notice if those lines were wrong. The difference is the whole subject. A test suite exists to fail when the code is broken, and mutation testing checks that directly: break the code on purpose, in many small ways, and count how many of the breakages the suite catches.

This article runs a mutation tool on Go 1.25.6, on a package where line coverage is already 100%, and reports what it found. The tool is gremlins, version v0.6.0, installed with go install in the lab. The answer to the question in the title is yes, and the result also shows where the technique wastes your time.

The code and its tests

package grade

// Pass reports whether a score passes the 60 point threshold and is not capped.
func Pass(score int) bool {
	if score < 0 || score > 100 {
		return false
	}
	return score >= 60
}

// Bonus adds 5 points to scores of 90 and above, capped at 100.
func Bonus(score int) int {
	if score >= 90 {
		score += 5
	}
	if score > 100 {
		return 100
	}
	return score
}

package grade

import "testing"

func TestPassWeak(t *testing.T) {
	if !Pass(80) {
		t.Fatal("80 should pass")
	}
	if Pass(10) {
		t.Fatal("10 should fail")
	}
	if Pass(-1) {
		t.Fatal("-1 invalid")
	}
}

func TestBonusWeak(t *testing.T) {
	if Bonus(50) != 50 {
		t.Fatal("no bonus below 90")
	}
	if Bonus(98) != 100 {
		t.Fatal("cap at 100")
	}
}

Pass accepts a score from 60 to 100. Bonus adds 5 points to scores of 90 and above, capped at 100. The tests check a typical pass, a typical fail, an invalid negative score, a score with no bonus and one that hits the cap. Between them they execute every statement.

What the tool does

A mutation tool parses the source, finds places where a small change is possible, and for each one writes the changed program, runs the tests, and records the outcome. The usual mutators are:

A mutant is killed if some test fails, lived if all tests pass, timed out if the run hangs, and not covered if no test reaches the line. The headline number is test efficacy: killed divided by killed plus lived.

go install github.com/go-gremlins/gremlins/cmd/[email protected] 2>&1 | grep -v '^go: downloading'
G='gremlins unleash --workers 1 --timeout-coefficient 50 .'
echo "# line coverage of the weak tests"
go test -count=1 -cover . | sed -E 's/\t[0-9.]+s//'
echo
echo "# mutation run, weak tests"
$G 2>&1 | sed -E 's/\x1b\[[0-9;]*m//g' | grep -E 'LIVED|KILLED|Killed|efficacy|Mutator'
cat > boundary_test.go <<'EOF'
package grade

import "testing"

func TestBoundaries(t *testing.T) {
	for score, want := range map[int]bool{-1: false, 0: false, 59: false, 60: true, 100: true, 101: false} {
		if got := Pass(score); got != want {
			t.Errorf("Pass(%d) = %v, want %v", score, got, want)
		}
	}
	for score, want := range map[int]int{89: 89, 90: 95, 95: 100, 96: 100, 100: 100} {
		if got := Bonus(score); got != want {
			t.Errorf("Bonus(%d) = %d, want %d", score, got, want)
		}
	}
}
EOF
echo
echo "# mutation run, with boundary tests added"
$G 2>&1 | sed -E 's/\x1b\[[0-9;]*m//g' | grep -E 'LIVED|KILLED|Killed|efficacy|Mutator'

# line coverage of the weak tests
ok  	example.test/lab	coverage: 100.0% of statements

# mutation run, weak tests
       LIVED CONDITIONALS_BOUNDARY at grade.go:5:11
      KILLED CONDITIONALS_NEGATION at grade.go:5:11
       LIVED CONDITIONALS_BOUNDARY at grade.go:5:24
      KILLED CONDITIONALS_NEGATION at grade.go:5:24
       LIVED CONDITIONALS_BOUNDARY at grade.go:8:15
      KILLED CONDITIONALS_NEGATION at grade.go:8:15
       LIVED CONDITIONALS_BOUNDARY at grade.go:13:11
      KILLED CONDITIONALS_NEGATION at grade.go:13:11
       LIVED CONDITIONALS_BOUNDARY at grade.go:16:11
      KILLED CONDITIONALS_NEGATION at grade.go:16:11
Killed: 5, Lived: 5, Not covered: 0
Test efficacy: 50.00%
Mutator coverage: 100.00%

# mutation run, with boundary tests added
       LIVED CONDITIONALS_BOUNDARY at grade.go:5:11
      KILLED CONDITIONALS_NEGATION at grade.go:5:11
      KILLED CONDITIONALS_BOUNDARY at grade.go:5:24
      KILLED CONDITIONALS_NEGATION at grade.go:5:24
      KILLED CONDITIONALS_BOUNDARY at grade.go:8:15
      KILLED CONDITIONALS_NEGATION at grade.go:8:15
      KILLED CONDITIONALS_BOUNDARY at grade.go:13:11
      KILLED CONDITIONALS_NEGATION at grade.go:13:11
       LIVED CONDITIONALS_BOUNDARY at grade.go:16:11
      KILLED CONDITIONALS_NEGATION at grade.go:16:11
Killed: 8, Lived: 2, Not covered: 0
Test efficacy: 80.00%
Mutator coverage: 100.00%

The first run

The coverage line shows 100.0%. Gremlins then ran 10 mutants. Five were killed: every CONDITIONALS_NEGATION. Five lived: every CONDITIONALS_BOUNDARY. The efficacy was 50.00%.

The pattern is the finding. The tests catch a flipped condition, because flipping score >= 60 to score < 60 reverses the result for the typical values 80 and 10. They do not catch an off-by-one, because the inputs 80, 10, 50 and 98 are all far from every boundary. A >= that became > changes the answer only at exactly 60 or exactly 90, and the tests never use those values. This is the commonest weakness in hand-written tests, and a coverage tool cannot see it.

Fixing it with boundary values

The second half of the script adds a file with the values that sit on each boundary: -1, 0, 59, 60, 100 and 101 for Pass, and 89, 90, 95, 96 and 100 for Bonus. The efficacy rose to 80.00%, with Killed: 8, Lived: 2.

That is what a mutation report is for. It told us where to add tests, the added tests were boundary tests, and the numbers confirm the effect. Without it we would have shipped a package with full coverage and an unguarded >=. Compare this with the coverage article, which reaches the same conclusion by editing the source by hand.

The survivors that cannot be killed

Two mutants still live: score < 0 to score <= 0 on the first line of Pass, and score > 100 to score >= 100 in Bonus. Look at each. For Pass, a score of 0 reaches the next line either way and fails score >= 60, so the function returns false in both versions. For Bonus, a score of exactly 100 returns 100 whether the cap triggers or not.

These are equivalent mutants: changes that produce a program with identical behaviour on every input. No test can kill them, because there is no behaviour to detect. Every mutation tool has them, and they put a ceiling below 100% on the score. They also point at something worth a look: a boundary that does not matter is sometimes a sign the condition is redundant, and removing it simplifies the code.

The practical consequence is that you do not chase a score. You read the list of survivors, decide for each whether it is a real gap or an equivalent change, and write a test for the first kind. Ten mutants are easy to read. Two thousand are not.

Cost and how to keep it manageable

Mutation testing is expensive. The tool runs the suite once per mutant, so a suite that takes 30 seconds and a package with 500 mutants takes four hours. Control the cost:

• Run it on a package, not a repository. Choose the packages where logic is dense: parsing, pricing, state machines, permission checks. Skip generated code and glue.
• Run it on the change. Gremlins and similar tools can restrict to changed files; a pull request that touches grade.go mutates grade.go.
• Use the coverage data. The tool already skips code no test reaches and reports it as not covered, which saves work.
• Make the suite fast. A mutation run multiplies test time, so everything in the slow suite article pays off twice.
• Limit workers if timeouts appear. In the lab, the default settings produced TIMED OUT for most mutants, a sign that parallel runs on a small machine slowed the tests below the tool's timeout. --workers 1 --timeout-coefficient 50 fixed it. A timed out mutant is counted separately, and it is not a kill you can trust.

Run it nightly or weekly, not on every commit, and treat the report as a to-do list.

Reading the tool's flags

Gremlins has a small number of flags worth knowing, taken from its own help output.

Before the first real run, use --dry-run to count. A package with 400 mutants and a 20 second test run needs more than two hours. A package with 40 needs a quarter of one. The count tells you whether to run the whole thing, restrict it with --diff, or pick one file.

Triage: what to do with a survivor

Open each surviving mutant at its line and ask four questions in order.

1. Is the changed program behaviourally identical? If yes, it is equivalent. Leave it, or simplify the code if the condition was redundant.
2. Is the changed behaviour reachable in production? A mutant in an error branch that cannot occur is a lower priority than one in the main path.
3. What input would distinguish them? For a boundary mutant, the input is the boundary value and its neighbours. For an arithmetic mutant, any input where the operands differ.
4. Write that test and re-run. If the mutant is now killed, the new test has value. If it still lives, the reasoning in step 3 was wrong, which is worth knowing.

Spend your time on the first run's survivors in the code that matters. A report of 200 survivors is not a list of 200 missing tests; it is a map, and most of the red is in code you would not test further.

Comparing the techniques

They overlap on boundaries. The surviving mutants here were boundary mutants; a property test over integers from the range would have hit 60 and 90 within a hundred cases, and found the same gaps without a mutation tool. Which route you take depends on the code. For a pure function with a clear rule, a property is more direct. For code whose rules are scattered across conditions, mutation testing finds the weak spots without your having to name the rule.

Adopting it in a team

Start with one package that has real logic and good tests, so the first report is mostly signal. Run it once, fix the genuine gaps, and record the efficacy as a baseline. Add the run to a nightly job with --threshold-efficacy set a few points under the baseline, so a drop fails the job while ordinary variation does not. Share the first report in a review and discuss two or three survivors in detail, since most of the learning is in recognising which survivors are real. Do not require a score across the repository. A threshold on code that is mostly plumbing drives people to write assertion-free tests, and that is the problem the technique exists to expose.

Reading the numbers with care

The efficacy figure of 80 percent in the second run is a description of ten mutants in one small package, not a score to compare across projects. A package with two functions and ten mutants moves by ten points when one mutant changes sides, and a tool with different mutators would report different numbers for the same tests. Use the figure inside one project over time, and use the list of survivors, not the percentage, to decide what to do next.

Other tools

Gremlins is one option. go-mutesting (by zimmski, with maintained forks) is older and has a wider set of mutators, including statement removal, and works by generating mutated files and running the tests against each. Another approach is the one in the lab's coverage article: hand-written mutations for the three or four lines you are most worried about. For a small, important function, five manual mutations are quicker than configuring a tool. Both tools are described here from their documentation, and only gremlins was run.

A hand-made mutation, for one function

You do not need a tool to get the main benefit. Take the function you worry about most, list the operators and constants in it, and change them one at a time by hand: a >= to >, a 5 to 4, a + to -, a deleted return. After each edit run the package's tests, expect a failure, and revert. A function with six or eight such points takes ten minutes, and each edit that does not fail the tests is a missing assertion that you have just found without installing anything. The coverage article does this with perl and a one-line change. The tool's advantage is not intelligence but stamina: it will try every mutation in every file, repeatedly, without getting bored, and keep a record.

Limits and honest expectations

• It tests the tests, not the code. A suite that kills every mutant can still miss a requirement the code does not implement.
• The mutators are syntactic. They find missing assertions on operators and constants. They do not find a wrong algorithm.
• Survivors need judgment. Some are equivalent, some are in logging or defensive code that tests rightly ignore.
• A high score is not the goal. A team that writes tests to kill mutants produces assertions on implementation details. Use the report to find what you forgot to specify.
• Results vary with the machine. The timeout behaviour above was specific to this laptop and settings.

When it is worth it

Use mutation testing on a handful of packages where a silent logic error is expensive, and run it before a release or after a refactor. Skip it for code that is mostly plumbing or already covered by property tests and fuzzing, which probe the same boundaries from the input side; see property based testing. The most valuable discovery in this run was not the score, and it would not have appeared in any report that stopped at line coverage. It was seeing that 100% coverage and 50% efficacy describe the same test suite. The tool is documented in the gremlins repository, and the testing hub is the Go testing guide.