Stream JSON in Go With the Encoder and Decoder Types
Stream JSON in Go without loading it all: an iterator over a top-level array, NDJSON with a Decoder, element-by-element encoding and a live heap of 0.02 MB.
json.Unmarshal wants the whole document in memory as bytes, and then builds the whole result in memory as Go values. For an API response of fifty items that is the right trade. For an export of five million rows, a log file, or a bulk import posted by a customer, it means the memory your service needs scales with the largest input it will ever see, which is not a number you control.
Tested with Go 1.25.6, macOS on Apple Silicon (arm64)
Go's json.Decoder and json.Encoder work on streams, and with a few lines you can process an array, or a file of newline-delimited JSON, one element at a time. This article builds the three helpers, runs them, and measures what they save, because the saving is real but is not the one people expect. It follows the tour of marshaling and leads into large responses without running out of memory. Everything ran on Go 1.25.6.
The helpers
package main
import (
"encoding/json"
"fmt"
"io"
"iter"
)
// Array yields the elements of a top-level JSON array one at a time, never holding more than one in memory.
func Array[T any](r io.Reader) iter.Seq2[T, error] {
return func(yield func(T, error) bool) {
var zero T
dec := json.NewDecoder(r)
if tok, err := dec.Token(); err != nil {
yield(zero, err)
return
} else if d, ok := tok.(json.Delim); !ok || d != '[' {
yield(zero, fmt.Errorf("expected a JSON array, got %v", tok))
return
}
for dec.More() {
var v T
if err := dec.Decode(&v); err != nil {
yield(zero, err)
return
}
if !yield(v, nil) {
return
}
}
if _, err := dec.Token(); err != nil { // the closing ]
yield(zero, err)
}
}
}
// Lines yields the values of a newline-delimited JSON stream (NDJSON). A Decoder reads values back to back, so
// there is no line length limit and no bufio.Scanner buffer to outgrow.
func Lines[T any](r io.Reader) iter.Seq2[T, error] {
return func(yield func(T, error) bool) {
dec := json.NewDecoder(r)
for {
var v T
err := dec.Decode(&v)
if err == io.EOF {
return
}
if !yield(v, err) || err != nil {
return
}
}
}
}
// WriteArray encodes values as one JSON array, writing each element as it is produced.
func WriteArray[T any](w io.Writer, items iter.Seq[T]) error {
if _, err := io.WriteString(w, "["); err != nil {
return err
}
enc := json.NewEncoder(w)
first := true
for it := range items {
if !first {
if _, err := io.WriteString(w, ","); err != nil {
return err
}
}
first = false
if err := enc.Encode(it); err != nil { // adds a newline after each element, which is valid whitespace
return err
}
}
_, err := io.WriteString(w, "]")
return err
}Array[T] reads a top-level JSON array lazily. It asks the decoder for the first token and checks that it is [. Then it loops while dec.More() says another element follows, decoding each into a fresh T and handing it to the caller. When the loop ends it reads the closing ] token, which also surfaces a truncated document. It returns an iter.Seq2[T, error], so a caller writes for ev, err := range Array[Event](r) and checks err per element. Because the iterator is a range-over-func, breaking out of the loop makes yield return false and the helper stops reading: a caller that wants only the first match pays for nothing after it.
Lines[T] reads newline-delimited JSON (NDJSON, also called JSON Lines), where each line is a complete value. Decoding repeatedly from one Decoder reads values back to back and skips the whitespace between them, so blank lines are harmless. It returns on io.EOF and passes any other error to the caller.
WriteArray[T] goes the other way. It writes [, then each element through an Encoder, with commas between, then ]. The elements come from an iter.Seq[T], so the producer can be a database cursor, a channel adapter or a loop, and no slice of the whole result exists. The Encoder appends a newline after each element, which is valid whitespace in JSON and shows in the output below as ⏎.
What they did
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"iter"
"strings"
)
type Event struct {
ID int `json:"id"`
Kind string `json:"kind"`
Data string `json:"data"`
}
func events(n int) iter.Seq[Event] {
return func(yield func(Event) bool) {
for i := 0; i < n; i++ {
if !yield(Event{i, "click", strings.Repeat("x", 40)}) {
return
}
}
}
}
func main() {
var buf bytes.Buffer
if err := WriteArray(&buf, events(5)); err != nil {
panic(err)
}
fmt.Printf("array written element by element (%d bytes):\n%s\n", buf.Len(), strings.ReplaceAll(buf.String(), "\n", "⏎"))
sum := 0
for ev, err := range Array[Event](bytes.NewReader(buf.Bytes())) {
if err != nil {
panic(err)
}
sum += ev.ID
}
fmt.Println("\nstreamed the array back, sum of ids:", sum)
// Stopping early costs nothing: the iterator stops reading.
for ev := range Array[Event](bytes.NewReader(buf.Bytes())) {
fmt.Println("first element only:", ev.ID)
break
}
fmt.Println("\n== NDJSON")
nd := "{\"id\":1,\"kind\":\"a\"}\n{\"id\":2,\"kind\":\"b\"}\n\n{\"id\":3,\"kind\":\"c\"}\n"
for ev, err := range Lines[Event](strings.NewReader(nd)) {
fmt.Println("decoder:", ev.ID, ev.Kind, err)
}
fmt.Println("\n== a single line longer than bufio.Scanner's default limit")
long := fmt.Sprintf("{\"id\":1,\"data\":%q}\n", strings.Repeat("y", 70_000))
sc := bufio.NewScanner(strings.NewReader(long))
for sc.Scan() {
}
fmt.Println("Scanner error:", sc.Err())
n := 0
for ev, err := range Lines[Event](strings.NewReader(long)) {
n += len(ev.Data)
_ = err
}
fmt.Println("Decoder read", n, "bytes of data without a limit")
fmt.Println("\n== malformed input is reported where it happens")
for ev, err := range Array[Event](strings.NewReader(`[{"id":1},{"id":2},{"id":`)) {
fmt.Println(ev.ID, err)
}
for _, err := range Array[Event](strings.NewReader(`{"id":1}`)) {
fmt.Println(err)
}
_ = json.Valid
}array written element by element (376 bytes):
[{"id":0,"kind":"click","data":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}⏎,{"id":1,"kind":"click","data":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}⏎,{"id":2,"kind":"click","data":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}⏎,{"id":3,"kind":"click","data":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}⏎,{"id":4,"kind":"click","data":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}⏎]
streamed the array back, sum of ids: 10
first element only: 0
== NDJSON
decoder: 1 a <nil>
decoder: 2 b <nil>
decoder: 3 c <nil>
== a single line longer than bufio.Scanner's default limit
Scanner error: bufio.Scanner: token too long
Decoder read 70000 bytes of data without a limit
== malformed input is reported where it happens
1 <nil>
2 <nil>
0 unexpected EOF
expected a JSON array, got {Five events were written one at a time into a 376-byte array, and the reader streamed them back and summed their ids to 10. The second loop takes only the first element and breaks, which stops the reader.
The NDJSON input has a blank line in the middle, and the decoder yields three events anyway.
The long-line block shows a trap with the usual alternative. A bufio.Scanner splits input into lines but has a default maximum token size of 64 KiB, and a line with a 70,000-byte field made it stop with bufio.Scanner: token too long. The scanner can be given a bigger buffer, but then you own a limit that you must pick and that someone will exceed. The Decoder read the same line and counted its 70,000 bytes of data with no limit, because it reads values, not lines. (A limit is still worth having for input from outside; put it on the reader with io.LimitReader or http.MaxBytesReader, where it is explicit.)
The last block shows how errors surface. A truncated array yields the two complete elements, then an unexpected EOF error. An input that is an object rather than an array fails at the first token with expected a JSON array, got {. In both cases the elements before the problem were already delivered, which is the right behavior for a stream, and a caller should decide whether partial results are acceptable.
package main
import (
"bytes"
"testing"
)
func TestArrayRoundTrip(t *testing.T) {
var buf bytes.Buffer
if err := WriteArray(&buf, events(1000)); err != nil {
t.Fatal(err)
}
n, last := 0, -1
for ev, err := range Array[Event](&buf) {
if err != nil {
t.Fatal(err)
}
if ev.ID != last+1 {
t.Fatalf("out of order: %d after %d", ev.ID, last)
}
last = ev.ID
n++
}
if n != 1000 {
t.Fatalf("got %d elements", n)
}
}
func TestEmptyArrayYieldsNothing(t *testing.T) {
var buf bytes.Buffer
WriteArray(&buf, events(0))
if buf.String() != "[]" {
t.Fatalf("got %q", buf.String())
}
for range Array[Event](&buf) {
t.Fatal("nothing should be yielded")
}
}=== RUN TestArrayRoundTrip
--- PASS: TestArrayRoundTrip (0.02s)
=== RUN TestEmptyArrayYieldsNothing
--- PASS: TestEmptyArrayYieldsNothing (0.00s)
PASS
ok example.test/lab 1.822sThe tests write a thousand events, read them back through Array and check count and order, and check that an empty array is [] and yields nothing.
What streaming saves
The saving is memory, and it is easy to measure the wrong thing. Start with the standard benchmark, 100,000 events:
package main
import (
"bytes"
"encoding/json"
"testing"
)
var payload = func() []byte {
var buf bytes.Buffer
WriteArray(&buf, events(100_000))
return buf.Bytes()
}()
func BenchmarkUnmarshalWholeArray(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
var all []Event
if err := json.Unmarshal(payload, &all); err != nil {
b.Fatal(err)
}
sum := 0
for _, e := range all {
sum += e.ID
}
}
}
func BenchmarkStreamArray(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
sum := 0
for e, err := range Array[Event](bytes.NewReader(payload)) {
if err != nil {
b.Fatal(err)
}
sum += e.ID
}
}
}goos: darwin
goarch: arm64
pkg: example.test/lab
cpu: Apple M1 Max
BenchmarkUnmarshalWholeArray-10 18 64787139 ns/op 27551151 B/op 200036 allocs/op
BenchmarkStreamArray-10 18 62735340 ns/op 10135948 B/op 300012 allocs/op
PASS
ok example.test/lab 2.541sWhole-array Unmarshal took about 64.7 ms and allocated 27.5 MB over 200,036 allocations. Streaming took about 62.9 ms and allocated 10.1 MB over 300,012 allocations. Two things stand out. Time is the same, so do not stream to go faster. And the number of allocations went up by half, because each element is decoded into a fresh value and the iterator machinery adds some, while the bytes dropped by about two thirds. B/op is total allocated over the run, though, not memory held at once, so it does not show what matters either.
What matters is what is still reachable while the work is happening. The test below forces a garbage collection and reads the live heap, once after a whole-array Unmarshal and once halfway through a stream, on a 500,000-element, 39.9 MB document.
//go:build !race
package main
import (
"bytes"
"encoding/json"
"runtime"
"testing"
)
func live() uint64 {
runtime.GC()
var ms runtime.MemStats
runtime.ReadMemStats(&ms)
return ms.HeapAlloc
}
// TestLiveHeap measures what is still reachable, after a forced collection, at the moment the work is half done.
func TestLiveHeap(t *testing.T) {
var buf bytes.Buffer
WriteArray(&buf, events(500_000))
payload := buf.Bytes()
base := live()
var all []Event
json.Unmarshal(payload, &all)
whole := live() - base
runtime.KeepAlive(all)
all = nil
var midway uint64
n := 0
for range Array[Event](bytes.NewReader(payload)) {
n++
if n == 250_000 {
midway = live() - base
}
}
t.Logf("payload %.1f MB | live heap: whole array %.1f MB, streaming at element 250,000 %.2f MB",
float64(len(payload))/1e6, float64(whole)/1e6, float64(midway)/1e6)
}go test -run TestLiveHeap -v -count=1 . | grep -E "payload|^(ok|FAIL)" peak_test.go:40: payload 39.9 MB | live heap: whole array 48.9 MB, streaming at element 250,000 0.02 MB
ok example.test/lab 1.052sThe whole array held 48.9 MB of live data. The stream, at element 250,000, held 0.02 MB. The consumer never has more than one event, and the garbage collector reclaims the rest as it goes. That is the benefit: memory use stays flat as input grows. The forced collection matters: reading HeapAlloc while the program runs also counts garbage not yet collected, and my first attempt at this measurement, without it, reported a misleading 75 MB for the streaming case.
Note what the test cheats on: the payload itself is 39.9 MB in memory, because it is a byte slice. A real streaming consumer reads from a file, a socket or an HTTP request body, where the input is never fully resident either. The decoder keeps a small buffer.
Streaming in an HTTP handler
For a response, create the encoder on the ResponseWriter and write elements as they come from the database. Three details decide whether it works in practice.
The status is committed with the first byte. If row 80,000 fails, the client already has a 200 and half an array. A truncated array is invalid JSON, so a careful client notices, but it cannot tell why. Plan for this: for NDJSON, write a final line such as {"error":"..."} that clients are told to expect; for a JSON array, make sure the failure produces a document that fails to parse; and log the cause on the server. Do the checks that can fail cheaply (authorization, query validation, the first row) before writing anything.
Flush deliberately. Output sits in buffers. For a long export it can be good to flush every few hundred rows so the client sees progress and proxies do not time out; http.NewResponseController(w).Flush() does it. Measure before adding it, since flushing too often turns a stream into a lot of tiny writes.
Timeouts need care. A server-wide WriteTimeout kills long streams. Set a per-request deadline with ResponseController.SetWriteDeadline and extend it as you make progress, as in server timeouts.
On the receiving side the same pattern applies to uploads: decode a large request body element by element and write rows to the database in batches, with http.MaxBytesReader as the outer limit. Bulk endpoints that take many items in one request (bulk APIs) are a natural fit.
When not to bother
If the document is small and bounded, Marshal and Unmarshal are simpler, and the simplicity is worth more than the memory. If the consumer needs random access (to sort, to join, to look something up by key), you have to hold it anyway, and streaming only moves the allocation. Where the shape is a big object rather than an array, stream the part that is large and treat the rest normally. The Decoder.Token documentation shows how to walk objects.
The APIs are documented in the encoding/json package, and the iterator types in the iter package.
