Avoid Copying Between Strings and Byte Slices in Go

Go has two byte oriented types that look interchangeable and are not: string, which is immutable, and []byte, which is mutable. Converting between them has to copy, because if the conversion shared memory, a later write through the slice would change a string that is supposed to be constant. Programs that read from the network or files into []byte and then work with strings, or the reverse, can spend a surprising share of their time and allocations on these copies. This article measures the cost of each conversion, finds the places where the compiler quietly avoids it, and covers the unsafe escape hatch with the conditions under which it is correct.

It is a part of reducing allocations and sits with escape analysis, string concatenation and the strings Builder. The cornerstone is the Go performance guide.

The cases

A 520 byte string (a 13 byte phrase repeated 40 times) and the same bytes as a slice. Eleven operations, each in a benchmark that keeps or consumes the result in a way the compiler cannot remove:

package main

import (
	"fmt"
	"strings"
	"testing"
	"unsafe"
)

var (
	bsink []byte
	ssink string
	nsink int
	bol   bool
)

func main() {
	s := strings.Repeat("hello gopher ", 40)
	b := []byte(s)
	m := map[string]int{s: 1}
	small := "key"
	smallb := []byte(small)
	ms := map[string]int{"key": 7}

	cases := []struct {
		name string
		f    func(*testing.B)
	}{
		{"[]byte(s), 520 bytes, result kept", func(t *testing.B) {
			for t.Loop() {
				bsink = []byte(s)
			}
		}},
		{"string(b), 520 bytes, result kept", func(t *testing.B) {
			for t.Loop() {
				ssink = string(b)
			}
		}},
		{"string(b) == s (comparison)", func(t *testing.B) {
			for t.Loop() {
				bol = string(b) == s
			}
		}},
		{"m[string(b)] (map lookup)", func(t *testing.B) {
			for t.Loop() {
				nsink += m[string(b)]
			}
		}},
		{"for range []byte(s)", func(t *testing.B) {
			for t.Loop() {
				for _, c := range []byte(s) {
					nsink += int(c)
				}
			}
		}},
		{"len(string(b))", func(t *testing.B) {
			for t.Loop() {
				nsink += len(string(b))
			}
		}},
		{"string(smallb), 3 bytes, result kept", func(t *testing.B) {
			for t.Loop() {
				ssink = string(smallb)
			}
		}},
		{"ms[string(smallb)] (map lookup)", func(t *testing.B) {
			for t.Loop() {
				nsink += ms[string(smallb)]
			}
		}},
		{"unsafe.String(&b[0], len(b))", func(t *testing.B) {
			for t.Loop() {
				ssink = unsafe.String(&b[0], len(b))
			}
		}},
		{"unsafe.Slice(unsafe.StringData(s), len(s))", func(t *testing.B) {
			for t.Loop() {
				bsink = unsafe.Slice(unsafe.StringData(s), len(s))
			}
		}},
		{"strings.Clone(s)", func(t *testing.B) {
			for t.Loop() {
				ssink = strings.Clone(s)
			}
		}},
	}
	for _, c := range cases {
		bench(c.name, c.f)
	}
	fmt.Println(len(bsink) > 0, len(ssink) > 0, nsink > 0, bol)
}

[]byte(s), 520 bytes, result kept              68.6 ns/op       576 B/op      1 allocs/op
string(b), 520 bytes, result kept              65.3 ns/op       576 B/op      1 allocs/op
string(b) == s (comparison)                    12.4 ns/op         0 B/op      0 allocs/op
m[string(b)] (map lookup)                      23.0 ns/op         0 B/op      0 allocs/op
for range []byte(s)                          1100.3 ns/op         0 B/op      0 allocs/op
len(string(b))                                  2.1 ns/op         0 B/op      0 allocs/op
string(smallb), 3 bytes, result kept           11.2 ns/op         3 B/op      1 allocs/op
ms[string(smallb)] (map lookup)                 6.9 ns/op         0 B/op      0 allocs/op
unsafe.String(&b[0], len(b))                    2.1 ns/op         0 B/op      0 allocs/op
unsafe.Slice(unsafe.StringData(s), len(s))          2.1 ns/op         0 B/op      0 allocs/op
strings.Clone(s)                               69.5 ns/op       576 B/op      1 allocs/op
true true true true

package main

import (
	"flag"
	"fmt"
	"sort"
	"testing"
)

func init() {
	testing.Init()
	flag.Set("test.benchtime", "200ms")
}

// bench runs f five times and prints the median, which damps the noise of a laptop.
func bench(name string, f func(b *testing.B)) {
	const runs = 5
	ns := make([]float64, 0, runs)
	bs := make([]int64, 0, runs)
	as := make([]int64, 0, runs)
	for i := 0; i < runs; i++ {
		r := testing.Benchmark(f)
		ns = append(ns, float64(r.T.Nanoseconds())/float64(r.N))
		bs = append(bs, r.AllocedBytesPerOp())
		as = append(as, r.AllocsPerOp())
	}
	sort.Float64s(ns)
	sort.Slice(bs, func(i, j int) bool { return bs[i] < bs[j] })
	sort.Slice(as, func(i, j int) bool { return as[i] < as[j] })
	fmt.Printf("%-38s %12.1f ns/op %9d B/op %6d allocs/op\n", name, ns[runs/2], bs[runs/2], as[runs/2])
}

What it cost

A conversion that is kept copies

The first three rows are the rule. A 520 byte conversion costs about 65 ns and one allocation of 576 bytes (520 rounded up to the allocator's size class of 576), whether it goes from string to bytes, bytes to string, or strings.Clone. The time is the allocation and the copy together. A 3 byte conversion costs 11.2 ns and 3 bytes. The cost scales with the length, so converting a large buffer (a request body, a file) is where it hurts, and converting once per request is usually fine while converting once per field in a hot loop is not.

The compiler skips the copy when it can

Four rows show 0 allocations because the conversion's result never escapes into something observable:

• Comparison. string(b) == s compares the bytes in place. 12.4 ns for 520 bytes, essentially a memequal.
• Map lookup. m[string(b)] hashes the bytes in place for the lookup. 23.0 ns, 0 allocations. (Inserting is different: m[string(b)] = v must keep a string as the key, so it copies.)
• Range over bytes. for range []byte(s) iterates over the string's bytes directly. The loop's 1100 ns is the 520 iterations, not a copy. Ranging over the string with for i := 0; i < len(s); i++ gives the same bytes; use for _, r := range s only if you want runes.
• Length. len(string(b)) is folded to len(b).

Those are documented optimisations and they apply to specific syntactic forms. The small case shows how narrow they are: ms[string(smallb)] allocated nothing (6.9 ns), but ssink = string(smallb) allocated, because the string is stored. A function boundary can break it too. If you write func key(b []byte) string { return string(b) } and call m[key(b)], the conversion happens inside key and its result is returned, so it copies, unless the function is inlined and the compiler sees the pattern again. The pattern is "convert, then use immediately in one of the supported forms". Write it that way in hot code, and do not hide the conversion in a helper unless you have checked the result.

Some other forms also avoid the copy: switch string(b) { case "GET": ... }, concatenation like "prefix" + string(b) (the result is a new string anyway, so one copy not two), and passing to append([]byte(nil), s...) which copies once on purpose. When in doubt, run go build -gcflags=-m (escape analysis) or a benchmark with -benchmem.

The unsafe forms

Since Go 1.20 the standard way to view bytes as a string, or a string as bytes, without copying is:

• unsafe.String(ptr *byte, len int) string, e.g. unsafe.String(&b[0], len(b))
• unsafe.Slice(ptr *byte, len int) []byte, with unsafe.StringData(s) to get the pointer from a string.

Both cost 2.1 ns and nothing in the lab, which is the cost of the benchmark loop. They create a header over existing memory. That is exactly the aliasing the safe conversion prevents, and the safety conditions move to you:

1. For unsafe.String from a []byte: the bytes must not be modified for as long as the string is in use. If you pass the string to something that stores it (a map key, a cache) and later reuse the buffer, the stored string changes under it, which breaks map invariants and can corrupt programs in ways that are very hard to debug.
2. For unsafe.Slice from a string: you get a slice over memory that may be read only (string literals live in read only sections). Writing through it can crash the program with a fault. Treat it as read only.
3. Lifetime. The memory stays alive while either header does, so there is no use after free as in C, only the aliasing problems.

Use them where a measurement shows a hot path and the conditions are easy to state: a parser that hands out substrings of a buffer it owns and never reuses; a function that hashes a []byte key via a string API for one call. Libraries like strings.Builder do this internally for the reason given in its article: they never modify bytes already written. For everything else the safe conversion at 65 ns per 500 bytes is a cheap price for a program that cannot alias memory by accident.

Avoiding the problem at the design level

The best conversion is the one you do not make.

• Pick one type per pipeline. If data arrives as bytes and is parsed as bytes, use bytes.Index, bytes.Split and bytes.Equal and not strings equivalents.
• Use the dual APIs. strconv.ParseInt takes a string; strconv.AppendInt writes into a byte slice. bufio.Scanner.Bytes() returns the internal buffer without a copy (valid until the next Scan), while Text() converts. Calling Bytes() in a loop and comparing with bytes.Equal avoids a copy per line.
• Convert at the boundary, once. Read a field as bytes, convert it to a string when you store it, and keep it a string afterwards.
• Use []byte for building, string for keeping. A Builder or a byte slice for assembly, then one conversion for the result.
• Beware io.ReadAll followed by string(). That is two copies' worth of memory at once for a large body. Decode streams instead.

Checklist

• A kept conversion copies: about 65 ns and one allocation for 520 bytes.
• Comparisons, map lookups, ranges and len of a conversion do not allocate in current Go.
• Do not wrap those patterns in helper functions without checking the cost.
• unsafe.String, unsafe.Slice and unsafe.StringData are free and require that nothing mutates the shared bytes.
• Prefer one representation per pipeline and convert once at the boundary.

The conversions are specified in the Go specification and the unsafe functions in the unsafe package.