Build a Concurrent File Hasher in Go With Progress

Checksumming a large tree of files is a perfect concurrency exercise because it is simple to state and rewards getting the structure right. Each file is independent, so the work parallelizes cleanly. The expensive step is CPU bound (SHA-256), but each worker also waits on disk reads, so a fixed number of workers can keep both the disk and the cores busy. The pieces that make it a project and not a toy are the ones around the hashing: a walker that does not outrun the workers, progress reporting that does not tangle output, errors from unreadable files that do not stop the run, and cancellation that works in the middle of a large file.

This article builds it in stages with tests, and measures the speedup. It uses pipelines, worker pools, context and atomics, and belongs to the concurrency patterns cluster. Everything ran on Go 1.25.6 on an Apple M1 Max with standard library packages only. If you want to compute checksums without writing code, the SHA-256 hash generator and the file checksum tool run in the browser.

The shape

The program has four stages joined by channels:

1. A walker goroutine traverses the directory tree with filepath.WalkDir and sends the path of every regular file to a paths channel.
2. N workers receive paths, hash each file, and send a Result to a results channel.
3. A collector goroutine receives results and appends them to a slice. It is the only goroutine that touches that slice.
4. A reporter goroutine reads two atomic counters on a ticker and calls a progress callback.

The walker is unbounded in how fast it can find files, the workers are bounded, and the unbuffered paths channel makes the walker wait for them. That is the backpressure: the walker can never queue a million paths ahead of the hashing.

Progress needs a denominator. A percentage requires knowing the total size before hashing, which means walking twice or buffering the list. Walking is cheap compared to reading every byte, so the first pass here only counts files and sizes, and the second pass does the work. For a huge tree on slow storage you would count while hashing and report "files done" without a percentage.

Hashing one file

The unit of work reads a file in chunks and feeds a hash. io.Copy into a hash.Hash does exactly that with a 32 KiB buffer, and memory stays flat whether the file is 1 KB or 100 GB. The context check inside the loop makes cancellation work mid-file; a context-aware Reader wrapper turns a plain io.Copy into a cancellable one:

package main

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"io"
	"os"
	"sync/atomic"
)

// ctxReader fails reads once the context is done, so a huge file cannot hold up cancellation.
type ctxReader struct {
	ctx   context.Context
	r     io.Reader
	bytes *atomic.Int64 // bytes processed so far, shared with the progress reporter
}

func (c ctxReader) Read(p []byte) (int, error) {
	if err := c.ctx.Err(); err != nil {
		return 0, err
	}
	n, err := c.r.Read(p)
	c.bytes.Add(int64(n))
	return n, err
}

// hashFile returns the hex SHA-256 of the file at path.
func hashFile(ctx context.Context, path string, bytes *atomic.Int64) (string, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", err
	}
	defer f.Close()
	h := sha256.New()
	if _, err := io.Copy(h, ctxReader{ctx: ctx, r: f, bytes: bytes}); err != nil {
		return "", err
	}
	return hex.EncodeToString(h.Sum(nil)), nil
}

The defer f.Close() runs on every return path, including the early return on a read error. Errors from Close on a file opened only for reading carry no information worth acting on, so ignoring it here is deliberate (the rule is different for files you write: errors from Close). The counter increments by the bytes actually read, so progress reflects real work, not file counts.

The pipeline

The pipeline function wires the stages. Each goroutine has exactly one job, and each channel is closed by exactly one party:

package main

import (
	"context"
	"io/fs"
	"path/filepath"
	"slices"
	"sync"
	"sync/atomic"
	"time"
)

type Result struct {
	Path string
	Sum  string
	Err  error
}

type Progress struct {
	Files, TotalFiles int64
	Bytes, TotalBytes int64
}

// Count walks the tree once to learn how many files and bytes there are.
func Count(root string) (files, bytes int64) {
	filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
		if err == nil && d.Type().IsRegular() {
			if info, err := d.Info(); err == nil {
				files++
				bytes += info.Size()
			}
		}
		return nil
	})
	return
}

// HashTree hashes every regular file under root with the given number of workers. It calls
// onProgress about every 20 ms and returns the results sorted by path.
func HashTree(ctx context.Context, root string, workers int, onProgress func(Progress)) []Result {
	totalFiles, totalBytes := Count(root)
	var filesDone, bytesDone atomic.Int64

	paths := make(chan string) // unbuffered: the walker waits for a free worker
	results := make(chan Result)

	// 1. The walker.
	go func() {
		defer close(paths)
		filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
			if err != nil || !d.Type().IsRegular() {
				return nil // unreadable directories and non-files are skipped
			}
			select {
			case paths <- path:
				return nil
			case <-ctx.Done():
				return ctx.Err() // stops the walk
			}
		})
	}()

	// 2. The workers.
	var wg sync.WaitGroup
	for range workers {
		wg.Go(func() {
			for path := range paths {
				sum, err := hashFile(ctx, path, &bytesDone)
				filesDone.Add(1)
				select {
				case results <- Result{Path: path, Sum: sum, Err: err}:
				case <-ctx.Done():
					return
				}
			}
		})
	}
	go func() { wg.Wait(); close(results) }() // closed only after the last worker is done

	// 3. The reporter.
	stopReporter := make(chan struct{})
	reporterDone := make(chan struct{})
	go func() {
		defer close(reporterDone)
		t := time.NewTicker(20 * time.Millisecond)
		defer t.Stop()
		report := func() {
			if onProgress != nil {
				onProgress(Progress{filesDone.Load(), totalFiles, bytesDone.Load(), totalBytes})
			}
		}
		for {
			select {
			case <-t.C:
				report()
			case <-stopReporter:
				report() // one last report so the caller sees 100%
				return
			}
		}
	}()

	// 4. The collector runs in this goroutine, the only one that touches `all`.
	var all []Result
	for r := range results {
		all = append(all, r)
	}
	close(stopReporter)
	<-reporterDone

	slices.SortFunc(all, func(a, b Result) int {
		switch {
		case a.Path < b.Path:
			return -1
		case a.Path > b.Path:
			return 1
		}
		return 0
	})
	return all
}

Walk through the ownership. paths is closed by the walker's goroutine, which is the only sender. results is closed by a helper that waits for the WaitGroup, because no single worker knows when the others are done; this is the same close-after-wait structure as in merging channels. The collector is the function's own goroutine, so the slice needs no lock, and the final sort puts the output in path order regardless of which worker finished first. That sort is what makes the output a function of the input and not of the scheduler.

Every blocking send selects on ctx.Done(). Without that, cancelling the context would leave workers blocked sending results that nobody collects, or the walker blocked on a path nobody receives. The reporter does a final report after the collector finishes, so a progress display always ends at 100 percent.

Running it

The demo creates a temporary tree of 100 files of 2 MiB each in a few subdirectories, hashes it with one worker and then with a growing number, and compares each result with the single worker's. A combined digest over the sorted path:sum lines is a compact way to check that two runs produced identical output:

package main

import (
	"context"
	"crypto/sha256"
	"fmt"
	"math/rand/v2"
	"os"
	"path/filepath"
	"time"
)

func makeTree() string {
	root, _ := os.MkdirTemp("", "hasher")
	r := rand.New(rand.NewPCG(7, 7))
	buf := make([]byte, 2<<20)
	for i := range 100 {
		for j := range buf {
			buf[j] = byte(r.Uint32())
		}
		dir := filepath.Join(root, fmt.Sprintf("dir%d", i%5))
		os.MkdirAll(dir, 0o755)
		os.WriteFile(filepath.Join(dir, fmt.Sprintf("file%03d.bin", i)), buf, 0o644)
	}
	return root
}

func digest(results []Result) string {
	h := sha256.New()
	for _, r := range results {
		fmt.Fprintf(h, "%s:%s\n", filepath.Base(r.Path), r.Sum)
	}
	return fmt.Sprintf("%x", h.Sum(nil))[:16]
}

func main() {
	root := makeTree()
	defer os.RemoveAll(root)
	files, bytes := Count(root)
	fmt.Printf("tree: %d files, %d MiB\n", files, bytes>>20)

	var baseline string
	for _, workers := range []int{1, 2, 4, 8, 16} {
		reports := 0
		var last Progress
		start := time.Now()
		results := HashTree(context.Background(), root, workers, func(p Progress) { reports++; last = p })
		elapsed := time.Since(start)

		d := digest(results)
		if baseline == "" {
			baseline = d
		}
		fmt.Printf("workers %2d: %5v  digest %s (same as 1 worker: %v)  progress reports %d, final %d/%d files, %d/%d MiB\n",
			workers, elapsed.Round(time.Millisecond), d, d == baseline, reports,
			last.Files, last.TotalFiles, last.Bytes>>20, last.TotalBytes>>20)
	}
}

tree: 100 files, 200 MiB
workers  1: 114ms  digest a3d9a082df44f008 (same as 1 worker: true)  progress reports 6, final 100/100 files, 200/200 MiB
workers  2:  58ms  digest a3d9a082df44f008 (same as 1 worker: true)  progress reports 3, final 100/100 files, 200/200 MiB
workers  4:  31ms  digest a3d9a082df44f008 (same as 1 worker: true)  progress reports 2, final 100/100 files, 200/200 MiB
workers  8:  24ms  digest a3d9a082df44f008 (same as 1 worker: true)  progress reports 2, final 100/100 files, 200/200 MiB
workers 16:  17ms  digest a3d9a082df44f008 (same as 1 worker: true)  progress reports 1, final 100/100 files, 200/200 MiB

The digest is identical for every worker count, which is the correctness check: concurrency changed how long it took, not what it computed. The progress callback fired repeatedly during the longer runs, and the final report always shows all 100 files and all 200 MiB. The times tell the performance story. One worker hashes about 200 MiB at the speed of a single core; adding workers divides that until the disk or memory bandwidth, or the core count, takes over. After the number of cores, more workers cannot help CPU-bound work and mostly add scheduling overhead (concurrency vs parallelism). On this machine, with the files in the page cache after the first run, the gain was close to the number of performance cores and then flat. On a cold cache, or a spinning disk, the disk dominates and extra workers can even hurt by making the reads random; the right number for slow disks is smaller.

Reading the numbers

One worker hashed 200 MiB in 114 ms, which is about 1.75 GiB per second. That is fast because this Mac has hardware SHA-256 instructions and the files were in the page cache, so the "disk" was memory. Two workers took 58 ms, four took 31 ms, eight 24 ms and sixteen 17 ms. The first doublings are close to ideal (1.97 times for two workers, 3.7 for four). Past the four performance cores the gains slow, and going from eight to sixteen workers still helped a little because the machine has ten cores and some of the work is the walker and the channel hand-offs. A rule of thumb for CPU-bound hashing is one worker per core, then measure with one or two more and one or two fewer; the best number is rarely the extreme.

The progress reporter fired six times during the one-worker run and once during the sixteen-worker run. That is the ticker working as intended: it reports at most every 20 ms, so a run that finishes in 17 ms produces only the final report. For a long run on real storage you would see hundreds of reports, and a CLI would render only the latest. Do not make the reporter faster to impress yourself; a human reads a progress line once or twice a second, and each report costs two atomic loads and a callback.

Treat these numbers as a demonstration of shape and not a benchmark. The same tree on a cold cache, or on a network disk, would show the reverse: the first run would be bound by I/O, extra workers would barely help, and past a point they would hurt by interleaving reads. If the speedup you see stops at two workers, the disk is the limit.

Showing progress in a terminal

The callback receives a Progress value and decides how to show it. For a command-line tool, the usual rendering is a single line that rewrites itself with a carriage return, drawn only when standard error is a terminal so that logs and pipes stay clean:

func isTerminal(f *os.File) bool {
	info, err := f.Stat()
	return err == nil && info.Mode()&os.ModeCharDevice != 0
}

func render(p Progress) {
	pct := 0.0
	if p.TotalBytes > 0 {
		pct = 100 * float64(p.Bytes) / float64(p.TotalBytes)
	}
	fmt.Fprintf(os.Stderr, "\r%5.1f%%  %d/%d files", pct, p.Files, p.TotalFiles)
}

// in main: var onProgress func(Progress); if isTerminal(os.Stderr) { onProgress = render }

Writing progress to standard error keeps standard output reserved for the checksums, so hasher dir > sums.txt works. The callback runs on the reporter goroutine only, so it needs no locking of its own.

From hashing to finding duplicates

Hashing every file is the honest way to find duplicates and also the slowest. Most files differ in size, and files of different sizes cannot be identical. A fast duplicate finder groups paths by size first and hashes only the groups with more than one member:

bySize := map[int64][]string{}
filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
	if err == nil && d.Type().IsRegular() {
		if info, err := d.Info(); err == nil && info.Size() > 0 {
			bySize[info.Size()] = append(bySize[info.Size()], path)
		}
	}
	return nil
})
for size, paths := range bySize {
	if len(paths) < 2 {
		continue // unique size: cannot have a duplicate
	}
	// hash only these paths with the worker pool, then group by checksum
}

On a typical home directory this skips the large majority of the bytes. A second filter hashes only the first and last megabyte and discards groups whose partial hashes differ, which avoids reading the middle of large unique files. Only files that survive both are hashed in full. The worker pool code is unchanged: you hand it fewer paths.

A single huge file does not parallelize

Concurrent hashing speeds up many files; it does nothing for one file, because SHA-256 processes blocks in sequence and each block depends on the previous state. If you need to hash a 50 GB file faster, the options are a faster algorithm, hardware instructions (the Go standard library uses them automatically where available), or a different construction: tree hashes such as BLAKE3 split the input into chunks that are hashed in parallel and combined. They produce different digests from SHA-256, so they are only useful when both ends agree to use them.

Errors per file, and what to do with them

A tree of any size contains something unreadable: a file that vanished between the walk and the open, a permission problem, a socket or a broken symlink. The design choice here is that a failure belongs to the file, not to the run. Each Result carries its own Err, the workers keep going, and the caller decides afterwards what a failure means. A verification tool exits non-zero if any result has an error; a backup tool logs them and continues; a deduplicator ignores unreadable files because it cannot prove anything about them.

Resist the temptation to abort the whole run on the first error with errgroup. That is the right behavior when the files form one unit (a release archive where a single bad file invalidates the set) and the wrong one for a tree where most files are independent. The same results slice supports both: counting errors afterwards is one loop. Two details are worth handling explicitly. Special files, such as named pipes and devices, would block a read forever, which is why the walker only sends regular files (d.Type().IsRegular()). And symbolic links are skipped by WalkDir, which does not follow them; if you need to follow links, you take on the cycle detection that the standard walker deliberately avoids.

Tests

The properties to pin: it finds every file, the sums match a direct computation, it handles an unreadable file without stopping, and cancellation returns promptly. The last one matters most for a tool people will interrupt with Ctrl+C:

package main

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"os"
	"path/filepath"
	"testing"
	"time"
)

func writeFiles(t *testing.T, files map[string]string) string {
	t.Helper()
	root := t.TempDir()
	for name, content := range files {
		path := filepath.Join(root, name)
		os.MkdirAll(filepath.Dir(path), 0o755)
		if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
			t.Fatal(err)
		}
	}
	return root
}

func TestSumsMatchADirectComputation(t *testing.T) {
	root := writeFiles(t, map[string]string{"a.txt": "alpha", "sub/b.txt": "beta", "sub/deep/c.txt": ""})
	results := HashTree(context.Background(), root, 4, nil)
	if len(results) != 3 {
		t.Fatalf("hashed %d files, want 3", len(results))
	}
	for _, r := range results {
		data, _ := os.ReadFile(r.Path)
		want := sha256.Sum256(data)
		if r.Err != nil || r.Sum != hex.EncodeToString(want[:]) {
			t.Errorf("%s: sum %s err %v", r.Path, r.Sum, r.Err)
		}
	}
}

func TestResultsAreSortedByPath(t *testing.T) {
	root := writeFiles(t, map[string]string{"z": "1", "m": "2", "a": "3", "q/b": "4"})
	results := HashTree(context.Background(), root, 8, nil)
	for i := 1; i < len(results); i++ {
		if results[i-1].Path >= results[i].Path {
			t.Fatalf("results out of order: %s before %s", results[i-1].Path, results[i].Path)
		}
	}
}

func TestUnreadableFileIsReportedNotFatal(t *testing.T) {
	root := writeFiles(t, map[string]string{"good": "ok", "bad": "secret"})
	if err := os.Chmod(filepath.Join(root, "bad"), 0); err != nil {
		t.Skip("cannot change permissions here")
	}
	results := HashTree(context.Background(), root, 2, nil)
	var failed, ok int
	for _, r := range results {
		if r.Err != nil {
			failed++
		} else {
			ok++
		}
	}
	if os.Geteuid() != 0 && (failed != 1 || ok != 1) { // root can read anything
		t.Fatalf("want 1 error and 1 success, got %d and %d", failed, ok)
	}
}

func TestCancelReturnsPromptly(t *testing.T) {
	files := map[string]string{}
	big := string(make([]byte, 4<<20))
	for _, n := range []string{"a", "b", "c", "d", "e", "f", "g", "h"} {
		files[n] = big
	}
	root := writeFiles(t, files)
	ctx, cancel := context.WithCancel(context.Background())
	cancel() // already cancelled: nothing should be hashed to completion
	start := time.Now()
	results := HashTree(ctx, root, 4, nil)
	if took := time.Since(start); took > time.Second {
		t.Fatalf("HashTree took %v after cancellation", took)
	}
	for _, r := range results {
		if r.Err == nil {
			t.Fatalf("%s completed despite a cancelled context", r.Path)
		}
	}
}

=== RUN   TestSumsMatchADirectComputation
--- PASS: TestSumsMatchADirectComputation (0.00s)
=== RUN   TestResultsAreSortedByPath
--- PASS: TestResultsAreSortedByPath (0.00s)
=== RUN   TestUnreadableFileIsReportedNotFatal
--- PASS: TestUnreadableFileIsReportedNotFatal (0.00s)
=== RUN   TestCancelReturnsPromptly
--- PASS: TestCancelReturnsPromptly (0.04s)
PASS
ok  	example.test/lab	1.279s

Note how TestUnreadableFileIsReportedNotFatal encodes an environmental caveat honestly: it skips where permissions cannot be changed, and relaxes the assertion when running as root, since root ignores file modes. A test that fails in a CI container for a reason unrelated to the code is a flaky test of its own kind (testing concurrent code). All four run under -race, which also checks that the counters, the collector and the sort do not conflict.

Where I would take it next

Hash duplicates cheaply. To find duplicate files, group by size first and hash only files that share a size; most files are unique by size alone and need no hash. A partial hash of the first and last megabytes narrows the rest.

Use a buffer pool. io.Copy allocates a 32 KiB buffer per call. With millions of files, io.CopyBuffer with a buffer from a sync.Pool removes that churn (sync.Pool).

Respect the disk. On a spinning disk, parallel reads of different files cause seeks. Limit workers to one or two there, or sort paths to read in directory order. The worker count should be a flag and not a constant.

Print a progress line, carefully. The callback receives a Progress; a CLI renders it with a carriage return on one line, and only when standard error is a terminal.

Switch the algorithm. SHA-256 is the safe default. BLAKE3 and xxHash are faster when you only need integrity or deduplication, and are outside the standard library. For verifying published releases, stay with SHA-256, which is what the checksums you compare against will use. The crypto/sha256 documentation and the io package documentation cover the two APIs used in the core of the program.