Download a File in Go With Progress and Resume Support
Download a large file in Go into a partial file, report progress, resume with Range and If-Range after a failure, and rename it into place when complete.
Downloading a file in Go is io.Copy from res.Body to a file, and that version is correct until the connection drops at 93 percent of a 4 GB file. Then it leaves a truncated file with a perfectly good name, and the next run either trusts it or starts over. A downloader that deserves the name does four extra things: it writes somewhere temporary, it shows progress, it resumes after an interruption, and it checks that what it has is complete before putting it where the user will look.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
This article builds that, tests it against a server that supports ranges and a server that does not, and shows the three status codes a resume can return. It is the client side of serving files with range requests, uses the timeouts from the HTTP client article, and sits in the web cluster that ends in a production HTTP server. Everything ran on Go 1.25.6 with the standard library only.
The protocol, from the client's side
The first request is an ordinary GET. If the response carries Accept-Ranges: bytes the server supports resuming. The client should remember the validator, the ETag (or Last-Modified if there is no tag), because a resume is only safe if the file is still the same one.
To resume, the client sends Range: bytes=N-, where N is the size of the partial file, together with If-Range: <the saved ETag>. The server answers one of three ways:
206 Partial ContentwithContent-Range: bytes N-M/TOTAL: the file is unchanged and the body is the missing tail. Append it.200 OKwith the whole file: either the server ignoresRange, orIf-Rangedid not match because the file changed. Either way the partial file is useless. Truncate and start again from zero.416 Range Not SatisfiablewithContent-Range: bytes */TOTAL: N is at or past the end. If N equals TOTAL, the download had actually completed before the crash; otherwise the partial file is wrong and should be discarded.
The server-side behavior for each of those is demonstrated in the range requests article. The rules are in RFC 9110 on range requests, and MDN's page on If-Range is the short version. A fourth rule closes the loop: on a 206, check that the start in Content-Range equals the offset you asked for before appending. A proxy that mangles ranges would otherwise corrupt the file silently.
The downloader
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
)
// progressWriter counts bytes and calls report whenever the percentage crosses another step.
type progressWriter struct {
w io.Writer
done int64
total int64
step int
last int
report func(done, total int64)
}
func (p *progressWriter) Write(b []byte) (int, error) {
n, err := p.w.Write(b)
p.done += int64(n)
if p.total > 0 && p.report != nil {
if pct := int(p.done * 100 / p.total); pct >= p.last+p.step || p.done == p.total {
p.last = pct - pct%p.step
p.report(p.done, p.total)
}
}
return n, err
}
// Download fetches url into dest. It keeps dest+".part" and dest+".etag" between attempts.
func Download(ctx context.Context, c *http.Client, url, dest string, wrap func(io.Writer) io.Writer, report func(done, total int64)) error {
part, etagFile := dest+".part", dest+".etag"
var offset int64
if fi, err := os.Stat(part); err == nil {
offset = fi.Size()
}
saved, _ := os.ReadFile(etagFile)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
if offset > 0 && len(saved) > 0 {
req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
req.Header.Set("If-Range", string(saved))
} else {
offset = 0
}
res, err := c.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
var total int64
flags := os.O_CREATE | os.O_WRONLY
switch res.StatusCode {
case http.StatusPartialContent:
var start, end int64
if _, err := fmt.Sscanf(res.Header.Get("Content-Range"), "bytes %d-%d/%d", &start, &end, &total); err != nil || start != offset {
return fmt.Errorf("server answered with the wrong range %q", res.Header.Get("Content-Range"))
}
flags |= os.O_APPEND
case http.StatusOK:
total = res.ContentLength // a full body: any partial file is discarded
flags |= os.O_TRUNC
offset = 0
if tag := res.Header.Get("ETag"); tag != "" {
os.WriteFile(etagFile, []byte(tag), 0o644)
}
case http.StatusRequestedRangeNotSatisfiable:
if i := strings.LastIndex(res.Header.Get("Content-Range"), "/"); i >= 0 {
if n, _ := strconv.ParseInt(res.Header.Get("Content-Range")[i+1:], 10, 64); n == offset {
return finish(part, dest, etagFile, offset)
}
}
os.Remove(part)
return errors.New("partial file did not match the server; removed it, run again")
default:
return fmt.Errorf("unexpected status %s", res.Status)
}
f, err := os.OpenFile(part, flags, 0o644)
if err != nil {
return err
}
var out io.Writer = f
if wrap != nil {
out = wrap(out)
}
pw := &progressWriter{w: out, done: offset, total: total, step: 25, report: report}
if total > 0 {
pw.last = int(offset*100/total) / pw.step * pw.step // do not re-announce what the resumed file already had
}
_, copyErr := io.Copy(pw, res.Body)
if err := f.Close(); err != nil && copyErr == nil {
copyErr = err
}
if copyErr != nil {
return copyErr // the .part file stays, so the next call can resume
}
return finish(part, dest, etagFile, total)
}
// finish checks the size and moves the file into place.
func finish(part, dest, etagFile string, want int64) error {
fi, err := os.Stat(part)
if err != nil {
return err
}
if want > 0 && fi.Size() != want {
return fmt.Errorf("incomplete: have %d bytes, expected %d", fi.Size(), want)
}
os.Remove(etagFile)
return os.Rename(part, dest)
}The shape is deliberate. The .part file only ever grows by append, and the real name appears through one os.Rename at the end, which is atomic on the same file system. An interrupted run leaves the .part and the saved tag, which is exactly the state the next run reads. progressWriter reports in steps of 25 percent here, computed from integer division, and also at completion; logging on every Write would print thousands of lines for a 4 GB file and slow the copy.
Two details are easy to miss. The Range and If-Range headers are sent only if both an offset and a saved validator exist; resuming without a validator is unsafe, because you cannot tell whether the file changed, so the code starts over. And the total is read from different places depending on the status: ContentLength of the response on a 200, and the number after the slash in Content-Range on a 206, since a 206's own Content-Length is only the size of the tail.
Four scenarios
The program serves a deterministic 1,000,000 byte file and runs the downloader through each case. A wrapper writer simulates a crash at exactly 400,000 bytes so the numbers are stable.
package main
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"time"
)
var errCrash = errors.New("simulated crash")
// crashWriter writes exactly limit bytes and then fails, like a dropped connection would.
type crashWriter struct {
w io.Writer
limit int
}
func (c *crashWriter) Write(b []byte) (int, error) {
if len(b) > c.limit {
n, _ := c.w.Write(b[:c.limit])
c.limit = 0
return n, errCrash
}
c.limit -= len(b)
return c.w.Write(b)
}
func main() {
content := bytes.Repeat([]byte("0123456789abcdef"), 62500) // 1,000,000 bytes
version := `"v1"`
var lastRange, lastIfRange string
rangeServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
lastRange, lastIfRange = r.Header.Get("Range"), r.Header.Get("If-Range")
w.Header().Set("ETag", version)
http.ServeContent(w, r, "data.bin", time.Time{}, bytes.NewReader(content))
}))
defer rangeServer.Close()
ignoresRange := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
lastRange = r.Header.Get("Range")
w.Header().Set("ETag", `"v1"`)
w.Write(content) // never looks at Range
}))
defer ignoresRange.Close()
dir, _ := os.MkdirTemp("", "dl")
defer os.RemoveAll(dir)
want := sha256.Sum256(content)
client := &http.Client{}
ctx := context.Background()
check := func(dest string) string {
b, err := os.ReadFile(dest)
if err != nil {
return "missing"
}
if sha256.Sum256(b) == want {
return fmt.Sprintf("%d bytes, checksum ok", len(b))
}
return fmt.Sprintf("%d bytes, checksum DIFFERENT", len(b))
}
partSize := func(dest string) int64 {
fi, err := os.Stat(dest + ".part")
if err != nil {
return -1
}
return fi.Size()
}
report := func(done, total int64) { fmt.Printf(" progress %3d%%\n", done*100/total) }
// 1. Crash at 400,000 bytes, then resume.
dest := filepath.Join(dir, "a.bin")
fmt.Println("1. server supports ranges")
err := Download(ctx, client, rangeServer.URL, dest, func(w io.Writer) io.Writer { return &crashWriter{w: w, limit: 400000} }, report)
fmt.Printf(" first attempt: %v; part file has %d bytes\n", err, partSize(dest))
err = Download(ctx, client, rangeServer.URL, dest, nil, report)
fmt.Printf(" second attempt: err=%v, sent Range=%q If-Range=%q\n result: %s\n", err, lastRange, lastIfRange, check(dest))
// 2. The file changes on the server between attempts.
dest = filepath.Join(dir, "b.bin")
fmt.Println("2. file replaced between attempts")
Download(ctx, client, rangeServer.URL, dest, func(w io.Writer) io.Writer { return &crashWriter{w: w, limit: 400000} }, nil)
version = `"v2"`
err = Download(ctx, client, rangeServer.URL, dest, nil, nil)
fmt.Printf(" resume with a stale ETag: err=%v, result: %s\n", err, check(dest))
version = `"v1"`
// 3. A server that ignores Range.
dest = filepath.Join(dir, "c.bin")
fmt.Println("3. server ignores Range")
Download(ctx, client, ignoresRange.URL, dest, func(w io.Writer) io.Writer { return &crashWriter{w: w, limit: 400000} }, nil)
err = Download(ctx, client, ignoresRange.URL, dest, nil, nil)
fmt.Printf(" err=%v, saw Range=%q, result: %s\n", err, lastRange, check(dest))
// 4. The crash happened after the last byte arrived but before the rename.
dest = filepath.Join(dir, "d.bin")
fmt.Println("4. part file already complete")
os.WriteFile(dest+".part", content, 0o644) // every byte is on disk, but the rename never happened
os.WriteFile(dest+".etag", []byte(`"v1"`), 0o644)
fmt.Printf(" part file has %d bytes before the retry\n", partSize(dest))
err = Download(ctx, client, rangeServer.URL, dest, nil, nil)
fmt.Printf(" err=%v, result: %s\n", err, check(dest))
}1. server supports ranges
progress 26%
first attempt: simulated crash; part file has 400000 bytes
progress 50%
progress 75%
progress 100%
second attempt: err=<nil>, sent Range="bytes=400000-" If-Range="\"v1\""
result: 1000000 bytes, checksum ok
2. file replaced between attempts
resume with a stale ETag: err=<nil>, result: 1000000 bytes, checksum ok
3. server ignores Range
err=<nil>, saw Range="bytes=400000-", result: 1000000 bytes, checksum ok
4. part file already complete
part file has 1000000 bytes before the retry
err=<nil>, result: 1000000 bytes, checksum okScenario one is the headline. The first attempt died at 400,000 bytes, leaving a part file of exactly that size, after reporting 26 percent (a report fires when a chunk crosses the 25 percent step, so the printed value is slightly past it). The second sent Range: bytes=400000- with the saved ETag in If-Range, received a 206, appended the missing 600,000 bytes, and reported 50, 75 and 100 percent, measured against the whole file and starting from the offset (an earlier version of my progress writer re-announced 40 percent on resume, which is the bug the last initialisation fixes). The finished file matches the checksum of the original.
Scenario two is why If-Range exists. After the server's ETag changed to "v2", the resume request did not match, the server answered 200 with the full new file, and the downloader truncated the part file and rewrote it. Without If-Range, the client would have appended the new file's tail to the old file's head: the correct size and a wrong checksum, which is the nastiest kind of corruption because nothing fails. The third scenario shows a server that never honours Range still works, just without the saving. And the fourth sets up a part file that already holds every byte, as if the process died just before the rename: the 416 branch notices that the offset equals the total and goes straight to the rename.
Details that matter in a real tool
Verify content as well as size. A matching size is a good sign, not a guarantee. If the publisher provides a checksum (a .sha256 file or a header such as Digest), hash the finished file and compare before renaming. Hashing while writing is cheaper than rereading: wrap the writer with io.MultiWriter(f, hasher), but remember that a resumed download has to hash the existing part first.
Cancel cleanly. Pass a context from the caller; Ctrl+C handling with signal.NotifyContext cancels the request, io.Copy returns, and the .part stays for next time. Do not delete it on error: keeping it is what makes resume possible.
Concurrency. For very large files from a fast server, several ranges downloaded in parallel and written with WriteAt can saturate a link a single stream cannot. It complicates resume (you need a map of completed ranges) and many servers throttle per connection anyway, so measure before building it.
Timeouts. A download is a long request, so Client.Timeout would cut it off. Leave it unset on the download client and rely on a transport with ResponseHeaderTimeout and a stall detector instead: a goroutine that cancels the context if no bytes arrive for 30 seconds. The progress writer is the natural place to reset that timer.
Disk space and permissions. Check free space when Content-Length is known, and create the .part file with the mode you want the final file to have. Rename is atomic only within one file system, so keep the part file in the destination directory, not in /tmp.
