Iterating Strings, Bytes, and Runes

A signup form enforces "usernames up to 20 characters" with one line:

func shortName(name string) string {
	if len(name) > 20 {
		return name[:20]
	}
	return name
}

For "Marguerite" it behaves. For "Zoësmörgåsbord" — or any name with non-ASCII letters — it can return a string that prints as Zoësmörgåsbord with a replacement character glued to the end, fails a round trip through a JSON encoder, or trips a validation regex that expects valid UTF-8. The line confused two different measurements.

A Go string is an immutable sequence of bytes, and for text those bytes are UTF-8 encoded. One character does not equal one byte: é is 2 bytes, 日 is 3, 😀 is 4, plain a is 1. So:

fmt.Println(len("Zoë"))    // 4 — bytes: Z, o, c3, af
fmt.Println(len("日本語"))   // 9 — three 3-byte characters
fmt.Println(len("héllo"))  // 6 — é costs two bytes

Indexing has the same altitude: s[i] is a byte (a uint8), never a character. s[0] for "日" is 0xE6, the first third of the character. Slicing s[:2] therefore cuts the character in half — the bytes form an invalid UTF-8 sequence, and anything that later decodes the string (printing, scanning, re-encoding) shows the corruption.

This is not a defect of the design; UTF-8 byte length is the measurement protocols, buffer limits, and database columns actually care about. len(s) <= 100 is a perfectly good payload-size check. The bug appears only when a byte budget is enforced against a character requirement, and the corrected validator measures in the unit the requirement named:

func shortName(name string) string {
	if utf8.RuneCountInString(name) <= 20 {
		return name
	}
	return string([]rune(name)[:20]) // decode, cut at a character, re-encode
}

RuneCountInString decodes without allocating a rune slice, and the []rune round trip guarantees the cut lands on a boundary — the conversion cost is trivial on a username. Two measurements, one rule: decide whether the spec said bytes or characters before writing the comparison. Byte truth and character truth are separate questions, and the habit of asking which one the requirement meant is what keeps every limit and every truncation correct — the next chapter builds on it with a cut that needs no rune slice at all.

Ranging over a string does not walk bytes blindly. The loop decodes one UTF-8 character per iteration and gives you the rune plus the byte offset where it starts:

s := "héllo"
for i, r := range s {
	fmt.Printf("%d: %q (%d bytes)\n", i, r, len(string(r)))
}
// 0: 'h' (1 bytes)
// 1: 'é' (2 bytes)
// 3: 'l' (1 bytes)
// ...

Note the offsets jump by the encoded size — i is a position in the byte string, not a character counter. That single fact gives you two correct truncation strategies.

Count while scanning, cut at the last byte offset:

func truncate(s string, max int) string {
	if max <= 0 {
		return ""
	}
	seen := 0
	last := 0
	for i, r := range s {
		if seen == max {
			return s[:i] // i is the byte offset where rune max would start
		}
		seen++
		last = i + len(string(r))
	}
	return s[:last] // whole string fits
}

Convert, slice by characters, convert back:

func truncateSimple(s string, max int) string {
	r := []rune(s) // decodes every character into its own slot
	if len(r) > max {
		r = r[:max]
	}
	return string(r) // re-encodes to UTF-8
}

[]rune(s) allocates one slot per character and string(runes) allocates again on the way back — fine for short strings and user-facing paths, wasteful inside a hot loop over megabytes of text. The scanning version allocates nothing and keeps byte positions, which matters when the truncated result must align with other byte-indexed data.

Two conversions worth separating: string(byte) builds a one-character string from a byte value (with the byte's numeric value as the code point — string(65) is "A"), while string(runes) encodes a rune sequence. Mixing them up is a classic source of mystery strings containing control characters. When you need characters, decode; when you need positions, stay in bytes; when you need both in one pass, range hands you both at once.

Production text handling is mostly a routing problem — each task has a natural altitude, and picking the wrong one is what produces mojibake and off-by-N bugs.

A byte loop is correct — and faster — when the job is structural. Validating that a token contains only [a-z0-9-], locating a colon in "host:port", or lowercasing ASCII protocol keywords never needs decoding:

func isSlug(s string) bool {
	if s == "" {
		return false
	}
	for i := 0; i < len(s); i++ {
		c := s[i]
		ok := c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '-'
		if !ok {
			return false
		}
	}
	return true
}

When one pass must decode but the input is untrusted — log lines, uploads, user bios — decode manually so invalid bytes cannot derail you:

func printable(s string) bool {
	for i := 0; i < len(s); {
		r, size := utf8.DecodeRuneInString(s[i:])
		if r == utf8.RuneError && size <= 1 {
			return false // invalid byte, or a lone encoded RuneError
		}
		if r < 0x20 && r != '\n' && r != '\t' {
			return false // control character
		}
		i += size // advance by the encoded width, never by 1
	}
	return true
}

DecodeRuneInString returns the rune and how many bytes it occupied. For malformed input it yields RuneError with size 1 — advancing by exactly one byte guarantees the loop terminates even on garbage, which is why the i += size pattern is the safe streaming form. (range over a string makes the same choice internally: undecodable bytes surface as one U+FFFD each.)

Finally, assemble output with strings.Builder rather than + in a loop — each + allocates a fresh string, while a builder appends into growing storage:

var b strings.Builder
b.Grow(len(in)) // capacity hint, same spirit as make([]byte, 0, n)
for i, r := range in {
	if r == '\t' {
		continue
	}
	b.WriteString(in[i : i+utf8.RuneLen(r)])
}
out := b.String()

Bytes for positions, runes for characters, one decoder for untrusted input — the rest of the standard library (strings and fmt) assumes you have already made that choice before you call it.