JSON Mode Outputs

Your ticket triage endpoint sends one message and receives:

Sure! Here is the classification:

```json
{"category": "billing", "priority": "high"}
```

json.loads(reply) fails immediately with JSONDecodeError: Expecting value: line 1 column 1. The object inside is perfectly valid — prose and a markdown fence wrapped around it are not. That is one defect. Strip the wrapper and a second one appears: nothing in the reply proves the keys are the ones you declared. The model may return severity instead of priority, drop category entirely, or nest everything under {"result": {...}}.

JSON mode sets response_format={"type": "json_object"} and constrains decoding so the completion parses as JSON. That removes the wrapper problem at the source. It guarantees nothing about field names, types, or enum values — JSON mode is a syntax guarantee, not a schema guarantee.

Models still decorate output in practice — providers differ in whether they hand back bare objects — so strip defensively either way:

import json

def extract_json(text: str) -> dict:
    text = text.strip()
    if text.startswith("```"):
        text = text.split("\n", 1)[1]     # drop the opener line, e.g. ```json
        text = text.rsplit("```", 1)[0]   # drop the closing fence
    return json.loads(text.strip())

split("\n", 1) cuts at the first newline only, so multi-line JSON survives intact. rsplit("```", 1) removes the last fence without touching backticks inside string values. Fed the reply above, the helper returns {"category": "billing", "priority": "high"} — parseable on the first try.

What each level of effort actually buys:

Stripping is a fallback, not a contract: when a provider already returns a bare object, the startswith check leaves it alone. The remaining gaps — absent keys, illegal values, truncation — are what the rest of this topic handles, and what Schema-Constrained Generation prevents before sampling happens.

With the wrapper gone, syntax-valid-but-wrong responses become the visible failure: {"category": "Billing", "priority": "URGENT"} parses cleanly and then trips a database CHECK constraint. A field validator plus a capped retry loop turns that into a fixable event instead of a bad row.

import json
from openai import OpenAI

ALLOWED = {
    "category": {"billing", "shipping", "account"},
    "priority": {"low", "med", "high"},
}

def validate(payload: dict) -> list[str]:
    errors = []
    for field, allowed in ALLOWED.items():
        if field not in payload:
            errors.append(f"missing key {field!r}")
        elif payload[field] not in allowed:
            errors.append(f"{field}={payload[field]!r} must be one of {sorted(allowed)}")
    return errors

client = OpenAI()

def classify(ticket: str, max_attempts: int = 3) -> dict:
    messages = [
        {"role": "system",
         "content": "Classify the ticket. Return JSON with keys category and priority."},
        {"role": "user", "content": ticket},
    ]
    for _ in range(max_attempts):
        reply = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            response_format={"type": "json_object"},
        ).choices[0].message.content

        messages.append({"role": "assistant", "content": reply})  # keep the attempt in view
        try:
            payload = json.loads(extract_json(reply))
        except json.JSONDecodeError as err:
            messages.append({"role": "user",
                             "content": f"Invalid JSON ({err}). Return only the object."})
            continue

        errors = validate(payload)
        if not errors:
            return payload
        messages.append({"role": "user",
                         "content": f"Fix these fields: {'; '.join(errors)}"})
    raise ValueError(f"no valid classification after {max_attempts} attempts")

Run it against a real ticket:

ticket: "I was charged twice for order 88213 and nobody has replied."
attempt 1 → {"category": "Billing", "priority": "URGENT"}
  user    → Fix these fields: category='Billing' must be one of ['account', 'billing',
            'shipping']; priority='URGENT' must be one of ['high', 'low', 'med']
attempt 2 → {"category": "billing", "priority": "high"}   returned

Three lines carry the design. Appending the failed assistant reply before the correction keeps the mistake in the transcript, so the model edits its own output instead of guessing what went wrong. The corrective message names the key, the value produced, and the legal set — "try again" adds no information, and since that string is prompt text, treat it like one: Prompt Versioning applies to retry copy exactly as it applies to system prompts. The max_attempts cap bounds spend per ticket, and exhaustion raises rather than returning the last broken payload, because a silently wrong classification reaches your dashboard while a loud failure reaches your logs.

Retries do not fix a truncated response. If parsing dies at the end of an otherwise well-formed object, look at finish_reason before adding another attempt.

Even with stripping and a repair loop in place, five classes of failure reach your code:

1. Valid JSON, wrong shape. A missing key, a wrapper object like {"result": {...}}, a list where an object was expected. Syntax was never the constraint.
2. Enum drift. urgent versus high versus URGENT. The response parses, the constraint rejects it. A growing synonym map treats each symptom; closing the set at generation time removes the class, which is the territory of Schema-Constrained Generation.
3. Silent truncation. max_tokens is too low, the object is cut mid-string, and parsing dies with an end-of-input error on otherwise valid JSON. Check finish_reason: if it reports length, raise the limit — another attempt will truncate the same way.
4. Content-short output. You asked for five order records and received three, all perfectly formed. Syntax-valid, content-short. Validate cardinality separately from types, or the gap reaches your dashboard unnoticed.
5. Provider divergence. Some deployments require the word "JSON" in the prompt, some return bare objects, some still decorate them with fences. Write the parsing layer to tolerate all three; a provider migration must not require rewriting your output path.

When JSON mode is the wrong tool. Two short fields are better served by a typed tool call: the argument schema is stricter than any validator you would write, and there is no round trip to repair anything — see Tool Definition Basics. Very large payloads pay constrained-decoding latency on every token while the model spends budget formatting instead of reasoning. And if you are extracting fields from text that already exists, delimiters beat JSON entirely: instruct the model to emit CATEGORY: billing and parse with a regex.

The parse-and-validate layer survives every upgrade you make to this pipeline. Move to schema binding and it becomes the last line that catches drift between the declared schema and your business rules — not a redundancy.