Semantic and Structural Chunking

A returns policy written in Markdown already says where its sections begin. ## Eligibility and ## Refunds are boundaries a human reader would use to look something up, and reading them costs one regex pass — no model, no similarity, no tuning. That is the premise of structural chunking: split where the document says to split.

import re

HEADING = re.compile(r"^(#{1,6})\s+(.+)$", re.M)

def sections(md: str) -> list[dict]:
    marks = list(HEADING.finditer(md))
    out, stack = [], []
    if marks and marks[0].start() > 0:
        out.append({"path": "(preamble)", "text": md[:marks[0].start()].strip()})
    for i, m in enumerate(marks):
        level = len(m.group(1))
        end = marks[i + 1].start() if i + 1 < len(marks) else len(md)
        stack = [h for h in stack if h[0] < level]          # drop deeper headings when a sibling starts
        stack.append((level, m.group(2)))
        out.append({"path": " > ".join(t for _, t in stack), "text": md[m.start():end].strip()})
    return [s for s in out if s["text"]]

Input: a policy with one #, two ##, and one ###.

Output:

'Returns Policy'                            16 chars
'Returns Policy > Eligibility'              85
'Returns Policy > Refunds'                  55
'Returns Policy > Refunds > Exceptions'     48

Three lines decide the behavior. The stack filter pops Eligibility before Refunds is pushed, so paths reflect the tree rather than the order headings appear — without it, every later section inherits a heading it does not belong to. Slicing from m.start() keeps the heading line inside the chunk, which means the embedding sees the section name and the citation can quote the path without a second lookup. Filtering empty sections removes the blank runs that pure layout leaves behind.

# Returns Policy
├── ## Eligibility         -> "Returns Policy > Eligibility"
├── ## Refunds             -> "Returns Policy > Refunds"
│   └── ### Exceptions     -> "Returns Policy > Refunds > Exceptions"
└── ## Shipping            -> "Returns Policy > Shipping"   (Exceptions popped)

The path is the payload: it travels with the chunk as metadata, becomes the citation line in the UI, and gives Metadata Filtering in Retrieval a key to filter on. Fixed windows (Fixed-Size Chunking) cannot produce it — a character offset says nothing about which section a passage came from.

Most corpora arrive as PDF text, email bodies, or wiki exports with no heading markup at all. Structural chunking then falls back to the other half of its name: embed each sentence once at ingest, compare each sentence with its neighbor, and cut where agreement collapses (Cosine Similarity).

def breaks(sims: list[float], floor: float) -> list[int]:
    """sims[i] = similarity between sentence i and sentence i+1 (0-based)."""
    return [i + 1 for i, s in enumerate(sims) if s < floor]

sims = [0.81, 0.78, 0.34, 0.77, 0.29, 0.83]
print(breaks(sims, 0.50))

Output: [3, 5]

sentences    s1 ----- s2 ----- s3 // s4 ----- s5 // s6
sim( i, i+1)   .81     .78     .34    .77     .29    .83
                             ^cut                ^cut

0.34 says sentences 3 and 4 stopped talking about the same thing, so the chunk ending with sentence 3 closes there and a new one opens at index 3. The detector is pure arithmetic over numbers you already computed; every sentence was embedded exactly once, and only at ingest — a query never pays for this.

A fixed floor is a starting point, not a policy. Corpora differ in baseline similarity: the same 0.50 shatters a FAQ where every answer resembles every question, and never cuts a Q&A set whose sentences are all loosely related. Production code picks the cut relative to the document's own distribution — cut where similarity falls below, say, the 20th percentile of sims — so the threshold adapts to the corpus instead of being tuned once and forgotten.

Two practical constraints ride along. Sentence segmentation must run before similarity, and it is wrong often enough to matter: e.g., 3.5, and quoted strings split "sentences" mid-thought, placing dips inside statements rather than between topics. And embedding every sentence is a real ingest cost — budget it as part of re-indexing, not as part of the query path.

The generic failure is applying one splitter to everything. Each format marks its units differently, and a rule that ignores the format does not merely chunk badly — it chunks at places the document never intended.

Whatever rule runs, measure the result before trusting it. Token counts per chunk, printed as a distribution, expose the problem that averages hide:

import statistics

sizes = [210, 225, 190, 240, 1900]                      # tokens per chunk
print(statistics.median(sizes), max(sizes),
      round(max(sizes) / statistics.median(sizes), 1))

Output: 225 1900 8.4

One chunk is 8.4x the median. At k=10 it consumes a fifth of a 10,000-token retrieval budget by itself, and because it contains more terms it matches more queries — the oversized chunk is over-represented in results, not only in storage. The fix is a bound in both directions: sub-split anything above a max token count with the fixed-window pass, and merge anything below a minimum into its predecessor.

Failure modes worth checking on every new corpus:

• Uncapped sections — a single heading spanning 6,000 tokens bypasses every chunk-size rule you configured.
• One rule for all formats — clause numbering and speaker turns are invisible to a heading regex.
• Sentence splitting on code — e.g. and string literals land dips inside statements.
• Ingest cost ignored — embedding every sentence at re-indexing is a batch job, not a query-path cost.
• A floor tuned on one corpus — the 20th-percentile dip threshold that works for FAQs shatters legal text.