Runes Bytes and Unicode in Go Without the Confusion

Bytes are what a computer stores. Runes are what a person might call letters. Go gives you both, a type for each, and a string that is secretly the first one while looking like the second. The confusion comes from the fact that for English text you can ignore the difference and everything works, and then one name with an accent in it shows up in production. This article sorts out the vocabulary, shows the arithmetic of UTF-8 with real output, and ends on the case that even runes do not solve. It follows strings as byte sequences and is part of the Go basics path.

Three words that are easy to mix up

A code point is a number assigned to a character by Unicode, written U+00E9 for é. A rune is Go's name for a code point, stored in an int32. A byte is eight bits, and UTF-8 is the encoding that turns code points into bytes. UTF-8 uses one byte for the 128 ASCII characters and two, three or four bytes for everything else, which is why a string's length in bytes and its length in characters can differ.

In Go source, 'a' is a rune literal, an untyped constant that takes the type of whatever you assign it to, and its default type is rune. "a" is a string. Mixing the two is a compile error, because they are different kinds of thing.

Measuring the same text three ways

Take a string with each width in it and ask the three questions that people confuse:

package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	s := "aé世😀"

	fmt.Println("bytes:", len(s))
	fmt.Println("runes:", utf8.RuneCountInString(s), len([]rune(s)))

	for i, r := range s {
		fmt.Printf("offset %d  %c  U+%04X  %d bytes\n", i, r, r, utf8.RuneLen(r))
	}

	r, size := utf8.DecodeRuneInString(s[1:])
	fmt.Printf("%c %d\n", r, size)
}

bytes: 10
runes: 4 4
offset 0  a  U+0061  1 bytes
offset 1  é  U+00E9  2 bytes
offset 3  世  U+4E16  3 bytes
offset 6  😀  U+1F600  4 bytes
é 2

The text has ten bytes and four runes. The loop prints the byte offset of each rune, which are 0, 1, 3 and 6: the offsets are not consecutive because the characters are different widths. utf8.RuneLen tells you how many bytes a rune needs, and utf8.DecodeRuneInString reads one rune from the front of a string and tells you how far it advanced, which is the primitive that everything else is built from. The %c verb prints the character, %04X prints its code point in hex, and the fmt verbs reference lists the rest.

Cutting a string in the wrong place

The classic bug is truncating by bytes. This is how you produce invalid UTF-8 from valid input:

package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	s := "日本語のテキスト"

	cut := s[:4]
	fmt.Printf("%q valid=%t\n", cut, utf8.ValidString(cut))

	runes := []rune(s)
	good := string(runes[:4])
	fmt.Printf("%q valid=%t\n", good, utf8.ValidString(good))
}

"日\xe6" valid=false
"日本語の" valid=true

Four bytes is a character and a third. The first line prints a string ending in an escape sequence for a partial character and reports valid=false. Convert to []rune, cut there, and convert back, and you get four whole characters. The conversion allocates a slice of four bytes per rune, so for a large string, or a hot loop, step through with utf8.DecodeRuneInString and remember the byte offset where you want to stop. A cheap check is utf8.RuneStart(s[i]), which tells you whether byte i is the first byte of a character.

If you display a truncated title with an ellipsis, strings.Builder plus a rune counter is the version I would write, and I would validate the input first with utf8.ValidString.

Reversing, uppercasing and the other things that look easy

Reversing a string by byte index scrambles any multi-byte character, because each character's bytes get reversed too. Reverse the runes:

package main

import (
	"fmt"
	"strings"
	"unicode"
)

func reverse(s string) string {
	r := []rune(s)
	for i, j := 0, len(r)-1; i < j; i, j = i+1, j-1 {
		r[i], r[j] = r[j], r[i]
	}
	return string(r)
}

func main() {
	fmt.Println(reverse("héllo, 世界"))
	fmt.Println(strings.ToUpper("straße ñ"), strings.ToTitle("dž"))

	letters := 0
	for _, r := range "Go 1.25 está listo!" {
		if unicode.IsLetter(r) {
			letters++
		}
	}
	fmt.Println(letters)
}

界世 ,olléh
STRAßE Ñ Dž
11

reverse is correct for code points and wrong for graphemes, which comes next. strings.ToUpper applies Unicode case mapping, so ñ becomes Ñ, and the German ß stays as it is under simple case mapping, which tells you that uppercase is not always a one-to-one letter swap. The unicode package answers category questions, IsLetter, IsDigit, IsSpace, IsUpper, for any rune. Use it instead of comparing against 'a' and 'z' ranges, which only work for English.

One character, several runes

Here is the case that runes do not fix. Unicode lets you write é as one code point (U+00E9) or as the letter e followed by a combining acute accent (U+0301). They look identical on screen and are different strings:

package main

import (
	"fmt"
	"unicode/utf8"

	"golang.org/x/text/unicode/norm"
)

func main() {
	composed := "café"
	decomposed := "café"

	fmt.Println(composed, decomposed, composed == decomposed)
	fmt.Println(utf8.RuneCountInString(composed), utf8.RuneCountInString(decomposed))

	fmt.Println(norm.NFC.String(decomposed) == composed)

	thumbs := "👍🏽"
	fmt.Println(utf8.RuneCountInString(thumbs), len(thumbs))
}

café café false
4 5
true
2 8

Both words render as the same four letters, and the comparison is false. Rune counts differ, 4 against 5. After normalizing the second into NFC form, the composed form that most keyboards produce, they compare equal. The standard library has no normalization package, so this is the golang.org/x/text/unicode/norm module, which the Go team maintains outside the standard library. If you store usernames, file names or search keys from different sources, normalize on the way in. Older macOS file systems stored names in decomposed form, so a name typed on one machine could fail to match the same name on another. It is a classic cause of "the file is right there" reports.

The last line shows the limit of runes. A thumbs up with a skin tone modifier is two code points, so RuneCountInString says 2 for one visible symbol, and a flag emoji is two regional indicator runes as well. The unit a reader perceives as one character is called a grapheme cluster, and Go's standard library does not segment them. If your product shows "280 characters remaining" and the count must match what users see, you need a library implementing Unicode's segmentation rules, and I would check whether the product definition really requires it before adding one. Counting runes is the honest compromise for most forms, and it is better than bytes.

Reading bytes from outside

When bytes arrive from a file or socket, you do not know they are UTF-8. The decision tree is short. If the format guarantees UTF-8, such as JSON, treat it as a string and let encoding/json complain about invalid input. If it is a legacy encoding, convert with golang.org/x/text/encoding before treating it as text. If it is binary, keep it as []byte and never let it pass through string and fmt where a stray byte can be replaced. bytes.Buffer and bufio.Scanner work on bytes and give you the choice of when to decode.

Two other places rune thinking matters: regular expressions and strings.Index. strings.Index returns a byte offset, so slicing with it is safe, but treating the result as a character count is not. And a pattern that uses . in the regexp package matches one rune, not one byte, which is what you want. A quick way to test a pattern against text with accents is the regex tester.

When bytes are the right level after all

Parsing protocols, scanning for a delimiter, hashing, comparing for equality and copying are all byte operations, and decoding runes in those places wastes time. strings.Index(s, ",") is correct on UTF-8 text because no multi-byte sequence contains the byte for a comma: UTF-8 was designed so that ASCII bytes never appear inside another character, which makes byte-oriented parsing of delimited text safe. Reach for runes when the unit is a character, and stay with bytes when the unit is a delimiter. The Go blog's article on strings explains the design, and the unicode/utf8 package documentation lists every function used above.