Chain-of-Thought Prompting

A triage assistant gets one instruction: reply with the ticket ID to handle next.

Input: T1 refund, SLA 4h, opened 3h ago.
       T2 outage, SLA 1h, opened 50m ago.
       T3 password reset, SLA 8h, opened 6h ago.

Direct answer:   "T3 — it has been waiting the longest."

With reasoning:  T1: 4h - 3h = 1h remaining
                 T2: 1h - 50m = 10m remaining
                 T3: 8h - 6h = 2h remaining
                 Smallest remaining budget is T2.
                 ANSWER: T2

The direct answer is fluent and wrong: time in the queue is not the quantity under comparison. Nothing in the input says "compute remaining SLA" — the deciding number exists only after arithmetic, so a model answering in one shot has to guess which comparison you wanted. Asking for the intermediate steps forces those comparisons to exist, and it moves the mistake into the trace where you can read it, instead of hiding it behind a plausible ticket ID.

The prompt change is one line in the user turn:

Think through it first. Show your work as a list.
End with a line exactly: ANSWER: <ticket id>

Parse only the last delimiter, because the trace itself may quote a hypothetical earlier answer:

def final_answer(text: str) -> str:
    for line in reversed(text.splitlines()):   # last match wins
        if line.strip().startswith("ANSWER:"):
            return line.split(":", 1)[1].strip()
    raise ValueError("no ANSWER line in response")

reversed() is deliberate. A reasoning trace can mention other candidate answers in passing, and the final line is the one the model committed to.

The trigger is a request for intermediate work: "think through it first," "show your work," "work it out step by step before answering." It works because reasoning becomes part of the task rather than a preamble the model may skip. Placement is a policy decision — put the trigger in the system prompt when every request needs reasoning, in the user turn when only some do.

Once the delimiter is specified, the response has a layout you can rely on:

+-----------------------------------+
| reasoning trace                   |
|   facts pulled from the input     |
|   intermediate computations       |
|   candidate answers considered    |
+-----------------------------------+
ANSWER: T2
       ^
       +-- the only line your code reads

Where a trace helps, and where it does not:

Reasoning moves accuracy most when the answer is computed rather than retrieved. On a lookup or a format conversion it adds tokens and a second failure surface: the model can reason correctly and then contradict itself in the ANSWER line, or drop the delimiter entirely and leave your parser raising on a production request.

Cost follows the same split. Trace length is output tokens on every call, and when traces are fed back as context later, they become input tokens too — Token Cost Estimation is where to price that before shipping. Instruction wording for the non-reasoning half of the task lives in Zero-Shot Instruction Writing.

Every traced answer pays for its trace twice: more output tokens, and more latency before the first byte of the real answer shows up in a streaming UI.

def trace_cost(out_price_per_m: float, calls: int) -> dict:
    direct_tokens = 60 * calls          # answer only
    traced_tokens = 650 * calls         # answer plus reasoning trace
    usd = lambda tokens: tokens / 1e6 * out_price_per_m   # per-million price -> per-token
    return {
        "direct_usd": round(usd(direct_tokens), 2),
        "traced_usd": round(usd(traced_tokens), 2),
        "ratio": round(traced_tokens / direct_tokens, 1),
    }

trace_cost(out_price_per_m=0.60, calls=10_000)
# {'direct_usd': 0.36, 'traced_usd': 3.9, 'ratio': 10.8}

At ten thousand calls the traced path emits about eleven times the output tokens — pocket change at toy scale, a real line item at product scale. The ratio line is the one to quote in a design review: it says the trace, not the answer, dominates the bill.

Route before you pay

Reasoning does not have to be all-or-nothing. A cheap label decides whether the answer must be computed at all, and only those requests buy a trace:

LABELLER = ("Label this task: compute if the answer needs arithmetic or several "
            "constraints applied; lookup if the answer is stated in the input. "
            "One word only.")

def dispatch(question: str) -> str:
    task = call(LABELLER, question).strip().lower()   # call() = plain chat, no trace
    if task == "compute":
        return final_answer(call(TRACED_PROMPT, question))
    return call(DIRECT_PROMPT, question)

On a mixed traffic profile this holds the traced fraction down to the cases that need it, and the label itself is a short completion. The rule behind the split: pay for a trace when the answer must be computed, skip it when the answer is already sitting in the input.

Failure modes to check on every traced prompt:

• Trace contradicts the answer — the steps point to T1, the ANSWER line says T2. The parser accepts it silently.
• Delimiter dropped — no ANSWER: line, so the request raises and the caller sees a 500.
• Trace leaked to users — the UI renders reasoning the product should never display; slice it out before sending.
• Reasoning on a lookup — a two-line API answer arrives as a paragraph of steps and latency doubles for no accuracy gain.

Whether the gain justifies the cost is an empirical question, not a stylistic one: run both variants over the same fixed inputs and compare. Prompt Iteration Workflow is the loop for that measurement, and when the task is about output shape rather than steps, Few-Shot Example Selection is the cheaper lever.