Few-Shot Example Selection

The invoice extractor had one instruction: return one line per line item, fields separated by |. It passed the test invoice and broke on half of production:

prompt: Extract the line items.

Sure! The invoice contains three items:
- 2 × M8 hex bolts at $0.14 each (part BOLT-M8)
- 1 × steel washer, 100-pack at $4.10 (part WSH-100)
- Shipping, $6.00

Every fact in that reply is right and every downstream parser is broken: no pipes, prose wrapped around the quantities, a shipping row mixed in with parts. Rewriting the instruction — "return ONLY pipe-separated lines, no commentary" — fixed some invoices and not others, because a longer rule was still competing with the model's habit of answering in prose.

Two examples ended the argument:

Input: 1 × Widget A @ $3.00
Output: Widget A|1|3.00

Input: 3 × Cable B @ $1.25, plus $5 shipping
Output: Cable B|3|1.25

The second example does two jobs at once: it demonstrates the pipe format, and it shows that shipping lines are dropped. That drop was never stated in the instruction, yet after the examples the model applied it consistently — including on invoices the examples never covered.

An instruction states a rule. An example demonstrates the rule in the exact shape you intend to receive back.

Four slots fill an example set. Two to five examples is the usual size, and every one has to earn its place:

Assemble the set in code so ordering and labels stay identical across runs:

EXAMPLES = [
    ("1 × Widget A @ $3.00", "Widget A|1|3.00"),
    ("3 × Cable B @ $1.25, plus $5 shipping", "Cable B|3|1.25"),
    ("2 × Gasket, no price listed", "Gasket|2|UNKNOWN"),   # the edge case
]

def build_prompt(query: str) -> str:
    shots = "\n\n".join(
        f"Input: {inp}\nOutput: {out}" for inp, out in EXAMPLES
    )
    return (
        "Extract line items as name|qty|price. Drop non-item lines. "
        "Use UNKNOWN when a price is absent.\n\n"
        f"{shots}\n\nInput: {query}\nOutput:"
    )

Two lines carry the weight. The comprehension renders every pair with identical Input: / Output: labels — inconsistent labels teach the model to imitate punctuation instead of structure. The trailing Input: {query}\nOutput: leaves the completion mid-shape, so the model continues the pattern instead of opening a fresh message with a preamble.

Running it:

Input: 4 × Nut B @ $0.05, handling $2.00
Output: Nut B|4|0.05

Rules for the set itself:

• Include one edge case. A set of clean inputs teaches nothing about ambiguous input, and ambiguous input is where production traffic lives.
• Keep styles uniform. If example two is chatty and example three is terse, the model must guess which pattern to copy — conflicting examples are worse than none.
• Never smuggle instructions into examples. A comment like # drop shipping inside an Output is a literal string the model may copy, not a rule. State the rule in prose; show its effect in the example.

Position changes how hard the model tries to match you. The reliable order is instructions first, examples second, live query last:

+----------------------------------------------+
| instructions  what rules govern this task    |
+----------------------------------------------+
| examples      2-5 Input/Output pairs         |
+----------------------------------------------+
| live query    the actual input, ending with  |
|               a half-finished Output:        |
+----------------------------------------------+

Examples placed before the instructions read as unexplained data — there is no rule for the model to attach them to. Examples placed after the live query arrive too late to shape the response. Because the query sits last, the shape of the final example is still the most recent thing the model saw when generation begins.

Consistency is the other half. One example that ends Thanks! and another that ends nothing teach two patterns, and the model alternates between them; in a log this shows up as "sometimes it works." When an example disagrees with an instruction, the example usually wins — it is concrete, and it sits closer to the output than the prose does.

Placement also decides cost. Every example is re-sent on every request, so a set that grew to twelve pairs is twelve pairs of tokens per call. If you find yourself adding pairs to fix a wording problem, the wording belongs in the instruction instead; Zero-Shot Instruction Writing covers rewriting a request to name inputs, limits, and failure policy rather than demonstrating them.

Test whether the set is doing anything at all. Run the same input twice — once with the pairs, once with the instruction alone:

query = "4 × Nut B @ $0.05, handling $2.00"

with_shots = run(build_prompt(query))
without    = run("Extract line items as name|qty|price. "
                 "Drop non-item lines. Use UNKNOWN when a price is absent.\n\n"
                 f"Input: {query}\nOutput:")

If both calls return the same parsable row, the instruction already carried the contract and the pairs are re-sent every request for nothing — cut them back to the edge case. If the instruction-only call is the one producing prose, at least one pair is weaker than the rule it is supposed to demonstrate, and replacing that pair beats appending a fifth.

Few-shot steers shape. When the task's difficulty is the reasoning rather than the format, Chain-of-Thought Prompting uses examples of traces instead of examples of outputs, and System Prompt Design keeps the standing rules — scope, refusals — out of the per-request prompt entirely.