How Tokenizers Work

A summarizer that passes on the English test set dies in its first production week. The trace shows 400 context_length_exceeded. The customer's message is 6,000 characters — shorter than the test messages that worked. Character count lied, because the model never sees characters: a tokenizer maps text to integer IDs first, and both the context window and the invoice are measured in those IDs.

Run a counter and the mismatch is immediate:

import tiktoken

enc = tiktoken.get_encoding('o200k_base')   # ships with the model, not with your code

samples = {
    'english': 'The quickest way to double your money is to fold it in half and wait.',
    'chinese': '把你的钱对折再等等，这是最快的翻倍方法。',
    'emoji': 'ship it 🚀🔥🎉',
}
for name, text in samples.items():
    print(name, len(text), '->', len(enc.encode(text)))

Input: the three strings above. Output:

english 69 -> 16
chinese 20 -> 14
emoji 11 -> 7

len(text) counts characters; len(enc.encode(text)) counts tokens — the vocabulary IDs the model will actually consume. The full picture, including the popular shortcut:

Under the hood the encoder is a table of roughly 200,000 vocabulary entries mapped to integers, and enc.decode(ids) reverses the mapping exactly for the text you encoded — the model's entire world is those integers. Punctuation joins the count too: wait. costs 2 tokens, so one-token-per-word is a second myth stacked on top of characters.\n\nThree consequences fall out of those rows. Non-English scripts and emoji consume more tokens per character than plain ASCII, so a string-length guard that protects English traffic still lets oversized CJK traffic through. Word boundaries are not token boundaries, so word counts do not translate either. And the ratio shifts with the content, so no constant repairs the estimate — you can only measure.

The measurement feeds two systems you build next: Context Window Budgeting counts these IDs against the model's limit, and Estimating Token Costs converts them into money. Both assume you know how the split is produced, which is the question byte-pair encoding answers.

A fixed vocabulary cannot contain every word, and splitting every word into raw bytes would make every sentence expensive. Byte-pair encoding finds the middle: start from characters, count adjacent pairs, merge the most frequent pair, repeat. The resulting merge list is the tokenizer — learned from frequency statistics during training and frozen afterward.

from collections import Counter

corpus = ['low'] * 5 + ['lower'] * 2 + ['newest'] * 6 + ['widest'] * 3
words = [list(w) + ['</w>'] for w in corpus]   # </w> marks the word boundary

def apply_merge(word, pair):
    out, i = [], 0
    while i < len(word):
        if i + 1 < len(word) and (word[i], word[i + 1]) == pair:
            out.append(word[i] + word[i + 1])   # the pair collapses into one token
            i += 2
        else:
            out.append(word[i])
            i += 1
    return out

merges = []
for _ in range(4):
    pairs = Counter()
    for w in words:
        pairs.update(zip(w, w[1:]))
    best = pairs.most_common(1)[0]              # most frequent adjacent pair wins
    merges.append(best)
    words = [apply_merge(w, best[0]) for w in words]
    print('merge', best)

Input: the 16-word corpus above. Output:

merge (('e', 's'), 9)
merge (('es', 't'), 9)
merge (('est', '</w>'), 9)
merge (('l', 'o'), 7)

Read the merges as a story. est occurs six times in newest and three times in widest, so it wins the first round; the end-of-word marker glues on next, and est</w> becomes one token. Encoding a new word replays those merges in training order:

def encode(word, merges):
    tokens = list(word) + ['</w>']
    for pair, _ in merges:
        tokens = apply_merge(tokens, pair)      # fixed order, not frequency order
    return tokens

print(encode('newest', merges))   # ['n', 'e', 'w', 'est</w>']
print(encode('lowest', merges))   # ['lo', 'w', 'est</w>']
print(encode('qzxv', merges))     # ['q', 'z', 'x', 'v', '</w>']

lowest never appeared in training, yet it costs three tokens because its suffix was learned from other words. qzxv matches no merge and stays at one token per character. The pattern generalizes: common English words map to one token because they appeared whole often enough to earn vocabulary slots (hello -> 1), while rare and unseen strings fall back to frequent pieces (antidisestablishmentarianism -> 6, asdfghjkl -> 4). Token count tracks familiarity with the training corpus, not string length — which is exactly why the character math in the previous chapter fails.

The loop is the whole mechanism: characters become frequent pairs, pairs become merged units, and each merge appends one line to the tokenizer's table. Two models with different merge lists count the same sentence differently, so the counter you run must match the model you call.

Character ratios and merge lists explain the split; production needs a number before the request leaves your process. Count locally with the encoding the target model uses:

import tiktoken

def fits(text, limit, encoding='o200k_base'):
    n = len(tiktoken.get_encoding(encoding).encode(text))
    if n > limit:
        raise ValueError(f'prompt is {n} tokens, limit is {limit}')  # fail here, not as a 400
    return n

Input: fits('assembled prompt text', 8000). Output: the token count, or a ValueError naming both numbers — raised while the user still sees a spinner rather than an error page.

The counter itself is cheap: tiktoken keeps the merge table locally, so the check runs inside your request path — about a millisecond for a 3,000-character prompt — and no prompt text leaves your process just to be counted.

The pre-check must cover everything that rides along: system prompt, conversation history, retrieved chunks, tool and schema definitions, and image parts. A check over the user's message alone approves requests the API later rejects.

Four counting failures seen in real services:

• chars / 4 in a guard. Passes English staging traffic, breaks on the first CJK or emoji-heavy message, as the earlier counts predicted.
• The wrong encoding. Vocabularies differ between model families and versions; counting with the previous model's tokenizer can drift by tens of percent. Pin the encoding to the model you call.
• Counting only the user's turn. System prompt, history, retrieval, tool schemas, and images all add tokens; every uncounted part shrinks the real margin.
• No local check at all. Each oversized request costs a full round trip, a 400, and often a blind retry that repeats the identical failure.

Counts also move when you edit the prompt: 200 tokens added to the system prompt subtract 200 from the history budget in Context Window Budgeting and multiply the bill in Estimating Token Costs. The ID sequence streaming back to the user in Streaming Responses is this same vocabulary — the count you make here is the count that matters there.