Build a Test Harness for a Go Command Line Tool Properly

A command line tool looks hard to test because it does everything at the edges: it reads os.Args, os.Stdin and files, writes to os.Stdout and os.Stderr, and ends with os.Exit, which kills the process, test included. Every one of those is an input or output that can be passed in instead. Once they are, the tool is an ordinary function and a table test covers it.

This project builds that test setup for a small word-frequency tool, wf, on Go 1.25.6. It has the pieces most command line programs need: flags, standard input, a file argument, error messages and exit codes. At the end there is one test that runs the compiled binary, and a list of what each layer is for. The same shape applies whether you write the flag parsing by hand, or use cobra, which is covered in a separate Cobra article.

The goal

By the end you have:

• a run function that contains the whole program and takes its environment as arguments;
• a table test of arguments, input and expected output and exit code;
• tests for the error paths, including usage output on stderr;
• one end to end test of the real binary;
• a rule for what goes in which layer.

Step 1: move the program into run

package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"os"
	"sort"
	"strings"
)

// run is the whole program with its dependencies passed in, so a test can call it without a process.
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
	fs := flag.NewFlagSet("wf", flag.ContinueOnError)
	fs.SetOutput(stderr)
	top := fs.Int("top", 3, "number of words to show")
	fold := fs.Bool("i", false, "ignore case")
	if err := fs.Parse(args); err != nil {
		if err == flag.ErrHelp {
			return 0
		}
		return 2
	}
	in := stdin
	if fs.NArg() > 0 {
		f, err := os.Open(fs.Arg(0))
		if err != nil {
			fmt.Fprintf(stderr, "wf: %v\n", err)
			return 1
		}
		defer f.Close()
		in = f
	}
	counts := map[string]int{}
	sc := bufio.NewScanner(in)
	sc.Split(bufio.ScanWords)
	for sc.Scan() {
		w := sc.Text()
		if *fold {
			w = strings.ToLower(w)
		}
		counts[w]++
	}
	words := make([]string, 0, len(counts))
	for w := range counts {
		words = append(words, w)
	}
	sort.Slice(words, func(i, j int) bool {
		if counts[words[i]] != counts[words[j]] {
			return counts[words[i]] > counts[words[j]]
		}
		return words[i] < words[j]
	})
	for i, w := range words {
		if i == *top {
			break
		}
		fmt.Fprintf(stdout, "%4d %s\n", counts[w], w)
	}
	return 0
}

func main() { os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr)) }

The signature is the design: run(args []string, stdin io.Reader, stdout, stderr io.Writer) int. Every input the program reads is a parameter, every output is a writer, and the exit status is the return value. main becomes a single line:

func main() { os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr)) }

That line is the only code that touches the real process, and it has no logic. os.Exit is called once, in main, so no test ever has to survive it. The pattern is explained in the os.Exit testing article.

A few details of the implementation are worth reading before the tests.

The flag set is local. flag.NewFlagSet("wf", flag.ContinueOnError) instead of the global flag.CommandLine. A global flag set is shared by every test in the process and calls os.Exit on an error by default. A local one with ContinueOnError returns the error, and fs.SetOutput(stderr) sends the messages where the test can read them.

Errors map to exit codes. A flag error returns 2, the convention for usage errors. A runtime problem, such as an unreadable file, returns 1 after printing wf: ... to stderr. -h returns 0 after printing usage. Documenting this in the code makes the test table the specification of the exit codes.

Output is sorted. The words are ordered by count descending, then alphabetically, so the output is deterministic. A program whose output depends on map order cannot be tested by comparing strings; fix the order in the code, not in the test.

Step 2: a table test of the whole behaviour

the cat and The dog
the end

package main

import (
	"bytes"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"testing"
)

func TestRun(t *testing.T) {
	cases := []struct {
		name     string
		args     []string
		stdin    string
		wantOut  string
		wantCode int
		wantErr  string
	}{
		{"counts stdin", []string{"-top", "2"}, "b a b c b a\n", "   3 b\n   2 a\n", 0, ""},
		{"ties sort alphabetically", nil, "z y x\n", "   1 x\n   1 y\n   1 z\n", 0, ""},
		{"ignore case", []string{"-i", "-top", "1"}, "Go go GO\n", "   3 go\n", 0, ""},
		{"file argument", []string{"testdata/sample.txt", "-top", "1"}, "", "", 0, ""},
		{"missing file", []string{"testdata/nope.txt"}, "", "", 1, "no such file"},
		{"bad flag", []string{"-wat"}, "", "", 2, "flag provided but not defined"},
		{"help", []string{"-h"}, "", "", 0, "Usage of wf"},
	}
	for _, c := range cases {
		t.Run(c.name, func(t *testing.T) {
			var out, errb bytes.Buffer
			code := run(c.args, strings.NewReader(c.stdin), &out, &errb)
			t.Logf("code=%d out=%q err=%.40q", code, out.String(), errb.String())
			if code != c.wantCode {
				t.Errorf("exit code = %d, want %d", code, c.wantCode)
			}
			if c.wantOut != "" && out.String() != c.wantOut {
				t.Errorf("stdout = %q, want %q", out.String(), c.wantOut)
			}
			if !strings.Contains(errb.String(), c.wantErr) {
				t.Errorf("stderr = %q, want it to contain %q", errb.String(), c.wantErr)
			}
		})
	}
}

// One test that runs the real binary, to cover main, os.Exit and the process boundary.
func TestBinary(t *testing.T) {
	bin := filepath.Join(t.TempDir(), "wf")
	if out, err := exec.Command("go", "build", "-o", bin, ".").CombinedOutput(); err != nil {
		t.Fatalf("build: %v\n%s", err, out)
	}
	cmd := exec.Command(bin, "-top", "1")
	cmd.Stdin = strings.NewReader("x y x\n")
	out, err := cmd.Output()
	t.Logf("binary stdout=%q err=%v", out, err)

	cmd = exec.Command(bin, "missing.txt")
	err = cmd.Run()
	if ee, ok := err.(*exec.ExitError); ok {
		t.Logf("binary exit code for a missing file: %d", ee.ExitCode())
	}
	_ = os.Stdout
}

go test -count=1 -v . 2>&1 | grep -vE '^(=== RUN|=== PAUSE|=== CONT|PASS|ok)' | sed -E 's/ \([0-9.]+s\)//; s#^ +[a-z_0-9]+\.go:[0-9]+: #    #'

    code=0 out="   3 b\n   2 a\n" err=""
    code=0 out="   1 x\n   1 y\n   1 z\n" err=""
    code=0 out="   3 go\n" err=""
    code=0 out="   2 the\n   1 The\n   1 and\n" err=""
    code=1 out="" err="wf: open testdata/nope.txt: no such file"
    code=2 out="" err="flag provided but not defined: -wat\nUsag"
    code=0 out="" err="Usage of wf:\n  -i\tignore case\n  -top int"
--- PASS: TestRun
    --- PASS: TestRun/counts_stdin
    --- PASS: TestRun/ties_sort_alphabetically
    --- PASS: TestRun/ignore_case
    --- PASS: TestRun/file_argument
    --- PASS: TestRun/missing_file
    --- PASS: TestRun/bad_flag
    --- PASS: TestRun/help
    binary stdout="   2 x\n" err=<nil>
    binary exit code for a missing file: 1
--- PASS: TestBinary

Each case is a row: the arguments, the standard input, the expected standard output, the expected exit code and a fragment the standard error must contain. The runner creates two bytes.Buffer values, calls run, and compares. It logs the code, the output and the start of stderr for every case, so the verbose run reads like a transcript:

• Counts stdin. -top 2 on b a b c b a gives 3 b and 2 a. The width-four count column is part of the format, which the exact string comparison pins.
• Ties sort alphabetically. z y x gives x, y, z, each with count 1. Without the tie-break the output would depend on map order.
• Ignore case. -i turns Go go GO into three counts of go.
• A file argument. The tool reads testdata/sample.txt instead of stdin.
• A missing file. Exit code 1 and wf: open testdata/nope.txt: no such file on stderr, nothing on stdout.
• A bad flag. Exit code 2, flag provided but not defined: -wat and the usage text.
• Help. Exit code 0 and Usage of wf: with each flag.

What the file case teaches

Look at the file case's output: 2 the, 1 The, 1 and. The arguments were testdata/sample.txt -top 1, and the result has three lines, not one. The -top 1 was ignored. Go's flag package stops parsing flags at the first argument that is not a flag, so everything after the file name is treated as a positional argument. Real users will type the flags last and be confused, and the tool should either document that or accept both orders.

This is the kind of thing a table test finds. The case was written with the natural order of arguments, the output was unexpected, and the test log made it visible. The decision is yours: keep the standard behaviour and say so in the usage message, or parse the remaining arguments again after the first positional. Write the expected output down in the table once you decide, so it stays decided.

Step 3: capture stderr and the exit code properly

The table's wantErr is a substring check, not an exact match. Error text from the standard library, such as no such file or directory, differs between operating systems and Go versions, and the flag package's usage text lists flags in sorted order with defaults. An exact match on those strings breaks for reasons unrelated to your code. Check the part you own, the wf: prefix and the fact that the message names the file, and let the rest vary.

Stdout is the opposite. It is the program's contract with whatever reads it, often another program, so compare it exactly. A change in whitespace is a breaking change for a script that parses the output. The wantOut field is exact whenever it is set.

When the output is large, store it in testdata and compare against a golden file with an -update flag. For a tool that prints a table or a report, a golden file is more readable than a string literal in the test.

Step 4: test the real binary once

run leaves a gap. main, the os.Exit call, the real os.Args and the process boundary are untested. TestBinary closes it. It builds the program into t.TempDir() with go build, runs it with exec.Command, sets Stdin to a reader, and checks the output and the exit status. The log shows:

• stdout 2 x for the input x y x with -top 1;
• exit code 1 when the file is missing.txt.

exec.ExitError.ExitCode() gives the status. This test takes about a second because of the compile, and one is enough: its job is to prove the wiring, not to repeat the table. Run it in the integration tier if the build is slow in your repository; the tag article shows how.

If you need to measure coverage of main and the process boundary, build the binary with go build -cover and set GOCOVERDIR, as in binary coverage. For tools that read the environment, set variables on the command with cmd.Env, not with t.Setenv, so the child gets a known environment and the test process is untouched. For tools whose behaviour depends on the terminal, test the non-terminal path (a pipe, as here) and treat the terminal path as a small manual check.

Making the table easier to grow

A table that works for seven cases gets awkward at forty. Three changes keep it manageable.

Load cases from files. For tools whose input and output are text, put each case in a directory under testdata/cases/name/ with args, stdin and stdout files, and have one loop walk the directory. Adding a case becomes adding a folder; the runner is twenty lines and never changes. The technique is the one in test cases from JSON, applied to a directory tree. Include an exit file for the expected status, defaulting to zero.

Share the runner. The loop that builds buffers, calls run and compares is the same for every command. Keep it in a helper, check(t, c), so a new command only adds data. Failure messages go through the helper, which means they are uniform: the case name, the arguments, the actual and expected output side by side, with t.Helper() so the line number is the caller's.

Separate behaviour from presentation. If the formatting of the output is volatile, test the data the tool computes through a function that returns it, and test the formatting in a smaller set of golden cases. Here, counting and sorting words could be a function returning a slice of word and count pairs, with a plain table of its own, while run only formats the first top entries. A change to the column width then touches one golden file and not forty table rows.

Exit codes as a contract

Scripts depend on exit codes more than on messages, so treat them as part of the interface. A small set is enough and conventional: 0 for success, 1 for a runtime failure, 2 for a usage error. Some tools reserve other values, for instance grep uses 1 for "no match" and 2 for an error, and diff uses 1 for "differences found". If your tool has an equivalent, document it and put a row in the table for each value.

Test the less obvious paths: what happens when stdout is closed (a broken pipe, as in wf | head -1), when stdin is empty, when the input is not valid UTF-8, and when the file is a directory. Each of these returns a status and a message in production. In a test, a closed stdout is a writer that returns an error from Write, and the table can assert that run exits non-zero instead of ignoring it. Ignoring write errors is the most common bug in command line tools, and it only shows when the output goes somewhere that fails.

Reading the output the way a user does

One more habit pays off. Run the compiled tool by hand once per release with the awkward inputs: no arguments, a huge file, a file with no trailing newline, a path with spaces. A table test checks what you thought of, and a few minutes at a terminal finds what you did not. When it does, add the case to the table so the next release does not rely on remembering to look.

The layers and what each is for

The rule that follows: put logic where it can be tested cheaply. If the binary test needs more than a couple of cases, the logic has leaked out of run.

Extending the setup

Signals and cancellation. A tool that handles SIGINT should take a context.Context in run, cancelled by signal.NotifyContext in main. Tests pass a context they cancel themselves, which is the approach in context cancellation tests. Sending a real signal to the binary is possible with cmd.Process.Signal, in the binary test.

Interactive input. Read prompts from the injected stdin, and write them to stdout, so a test feeds answers as a string. Detect a terminal with an injected function rather than calling isatty inside run.

Environment and configuration. Pass an env func(string) string or a map into run and use it instead of os.Getenv. Tests set values without touching the process environment, and can run in parallel.

Subcommands. A switch on args[0] that calls one run-style function per command. Each is a table of its own, and the top level has a table for dispatch and the unknown-command error.

Time and randomness. Inject a clock as in determinism.

Output colour and width. Take them as options, default them off when the output is not a terminal, and test both settings.

Limits

This setup checks what the tool does with its arguments, streams and exit status. It does not check that the tool behaves on a real terminal, that installers put it on the PATH, that it works on Windows paths and line endings, or that it is pleasant to use. Run the table on each operating system you ship, since path handling and line endings differ, and treat packaging as a separate test. The flag package is documented in the standard library reference, the process side in the os/exec reference, and the practice of testing programs through their input and output is in the testing guide.