Fixed-Size Chunking

A support bot answered "within 30" to the question how long do I have to return opened headphones? The sentence existed in the index: Opened headphones are returnable within 30 days of delivery if the seal is intact. Ingest had cut it in half, retrieval returned the half without its unit, and the model completed the sentence from its own priors.

Fixed-size chunking is the ingest rule that produced the cut: take size units of text, advance step units, repeat. The boundaries come from arithmetic, not from the document.

def naive(text: str, size: int = 60) -> list[str]:
    return [text[i:i + size] for i in range(0, len(text), size)]

Input:

policy = ("Refund window. Opened headphones are returnable within 30 days of "
          "delivery if the seal is intact. Shipping labels are issued after "
          "the return is approved.")
naive(policy)

Output:

['Refund window. Opened headphones are returnable within 30 da',
 'ys of delivery if the seal is intact. Shipping labels are is',
 'sued after the return is approved.']

Chunk 0 stops at 30 da; chunk 1 starts at ys. The query lands on chunk 0, the prompt receives a fragment with no unit, and nothing downstream — embedding, search, or generation — can tell that anything is missing.

|------------ size = 60 ------------|
                                     |------------ size = 60 ------------|
      <- no shared region: every character lives in exactly one chunk

Fixed windows earn their popularity honestly: one linear pass, no model calls, no grammar to get wrong, and identical input producing identical chunks, which keeps an index reproducible across re-runs. What they replace is regex sentence splitting, which falls over on 30 days. Mr. Smith, on numbered lists, and on any table exported to text. What they still get wrong is the boundary above — and the fix is a second parameter.

A character limit is a proxy for the thing that actually counts. The context window is measured in tokens, the bill is measured in tokens, and 4,000 characters are roughly 1,000 tokens in English but considerably more in Chinese or in a string full of emoji (How Tokenizers Work). Size chunks with the encoding the target model uses and the index cannot hand you a chunk that busts the retrieval budget (Context Window Budgeting).

Overlap is the second parameter: each window repeats the tail of the previous one, so a fact sitting on a cut is present in full somewhere.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")   # pin this to the model you call

def chunk(text: str, size: int = 40, overlap: int = 8) -> list[str]:
    if overlap >= size:
        raise ValueError("overlap must be smaller than size")
    ids = enc.encode(text)
    step = size - overlap                   # a stride, not an offset: this creates the shared region
    return [enc.decode(ids[i:i + size]) for i in range(0, len(ids), step)]

Input: the 77-token policy from the previous chapter, chunk(policy, 40, 8).

Output:

0 | Refund window. ... the buyer pays return shipping on non-defective
1 |  buyer pays return shipping on non-defective items. Refunds post ... missing a serial number are
2 | , and items missing a serial number are refused at the counter.

The same text cut with overlap=0 splits non-defective items. across two chunks. At step = 32 the second window opens eight tokens early, so the whole clause lands inside chunk 1 — and chunk 0 still ends mid-clause, which is harmless because the sentence exists elsewhere.

The table is the decision: characters are fine for throwaway scripts over ASCII, tokens are the only unit that predicts what the provider will accept, and sentences belong to the structure-aware pass covered in Semantic and Structural Chunking.

Overlap is not free. With size=40 and overlap=8, a 77-token document produces three windows storing 40 + 40 + 13 = 93 tokens — 21% more text than the source, re-embedded at ingest and re-billed whenever a chunk enters a prompt. Push overlap to 50% and the same document stores more than half again as much, while two copies of one sentence compete for top-k slots and can occupy rank 1 and rank 2 together.

overlap 0     |----A----|----B----|----C----|    cuts land wherever arithmetic puts them
overlap 20%   |---A---|---B---|---C---|          tail repeated once; boundary fact survives whole
overlap 50%   |-A-|-B-|-C-|                       index roughly doubles; near-duplicates crowd top-k

The second half of the fix happens when the chunk is created, not when it is queried. Every chunk should leave the chunker already knowing where it came from:

chunk = {
    "text": "Opened headphones are returnable within 30 days of delivery "
            "if the seal is intact.",
    "heading": "Returns Policy > Eligibility",
    "source": "returns-policy.md",
    "index": 0,
}

Heading and source become citation text, filter keys (Metadata Filtering in Retrieval), and the section label in the UI. A passage that reached the index without them cannot gain them later: the words alone do not say which section they came from, and re-deriving position from character offsets breaks the moment the source file is edited.

Failure modes to check on every chunking deployment:

• Overlap of zero — boundary sentences are only ever retrievable as fragments.
• Character sizing with no token check — English staging passes, production 400s on CJK traffic.
• Overlap past 30% — index cost climbs while top-k fills with copies of one paragraph.
• Chunks stored bare — nothing to cite, nothing to filter on, no section label to show.
• A 13-token tail — half a sentence that ranks on two content words and pollutes results.