Streaming Responses

The chat UI shows a spinner for 6.2 seconds and then drops a 400-token answer on screen in one lump. The model started writing at 800 ms; for 5.4 of those seconds the user watched nothing happen. Streaming does not make generation faster — it changes when the first token becomes visible.

That number has a name: time-to-first-token (TTFT), and it is what users perceive as "the system is thinking":

buffered:   [---- spinner 6.2s ----][ text lands at once ]
              ^ request              ^ user's TTFT = 6.2s

streaming:  [spinner 0.8s][ Id][emp][ot][ency][ means][ ...]
              ^ TTFT = 0.8s  ^ tokens keep arriving while the user reads

To see what actually arrives, watch the raw connection. "stream": true keeps the HTTP response open and turns it into text/event-stream: a sequence of data: frames, each holding one small JSON piece, separated by newlines:

POST /v1/chat/completions
{"model": "gpt-4o-mini", "stream": true, "messages": [...]}

<-- HTTP/1.1 200 OK
<-- content-type: text/event-stream

data: {"choices":[{"delta":{"role":"assistant"},"index":0}]}
data: {"choices":[{"delta":{"content":"Idem"},"index":0}]}
data: {"choices":[{"delta":{"content":"pot"},"index":0}]}
data: {"choices":[{"delta":{"content":"ency"},"index":0}]}
...
data: {"choices":[{"delta":{},"finish_reason":"stop"}],"index":0}]}
data: [DONE]

The same request through curl, which needs no SDK to make the point:

curl -N https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,
       "messages":[{"role":"user","content":"Define idempotency in one sentence."}]}'

Input: Define idempotency in one sentence.

Output (first frames): {"delta":{"role":"assistant"}...} → {"delta":{"content":"Idem"}} → {"delta":{"content":"pot"}}

Frame one carries the role and no text. Frame two starts the answer. The final frame carries finish_reason with an empty delta. Those three shapes are the whole protocol, and they are why a parser written against the buffered response crashes on frame one. Latency Profiling measures TTFT against your own budget; OpenAI SDK Essentials covers what streaming does to usage accounting.

No frame contains a sentence — each is a fragment, and the answer exists only after every fragment is concatenated in order. The client's job is a short loop with two guards that exist for specific crash reasons:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    stream=True,
    messages=[{"role": "user", "content": "Explain idempotency in one sentence."}],
)

parts = []
for chunk in stream:
    choice = chunk.choices[0] if chunk.choices else None   # terminal chunk: no choices
    if choice is None:
        continue
    parts.append(choice.delta.content or "")               # role-only frame: content None
    if choice.finish_reason:
        last_reason = choice.finish_reason

answer = "".join(parts)
print(last_reason, answer)

Input: Explain idempotency in one sentence.

Output:

stop Idempotency means retrying a request produces the same server-side effect as sending it once.

Why each guard exists:

• choice.delta.content or "" — the first frame sets delta.role and leaves content as None. Appending None raises TypeError on frame one; this is the most common first bug in streaming code.
• if chunk.choices else None — with stream_options={"include_usage": True} the final chunk arrives with an empty choices array and the usage block attached. Indexing [0] there raises IndexError.
• finish_reason arrives on a frame after the text ends. Read it instead of inferring completion from silence: stop means finished, length means the token cap cut the answer.

Anthropic's helper hides the frame bookkeeping behind an iterator of assembled fragments:

anthropic_client = Anthropic()      # key from $ANTHROPIC_API_KEY

with anthropic_client.messages.stream(
    model="claude-sonnet-4-5",
    max_tokens=512,
    messages=[{"role": "user", "content": "Explain idempotency in one sentence."}],
) as stream:
    for text in stream.text_stream:   # fragments, concatenated per content block
        render(text)

Output: the same sentence arriving as Idem → pot → ency fragments.

One invariant deserves an automated test: "".join(parts) must equal the non-streaming response for the same prompt and settings. When they diverge, the bug is in your guards — a dropped None, a frame counted twice after a reconnect — not in the model. The test scaffolding for exactly this kind of assertion is Testing Python AI Code.

A proxy kills the idle connection at 3.2 seconds and the user is holding half a sentence. Streaming trades one failure mode for another: instead of a slow blank wait, you now own a partially delivered response — and the recovery policy depends entirely on when the failure happened.

                    [stream opens]
                          |
         +----------------+----------------+
         | 401 / 400 / 429 / connect fail  |  first delta seen
         | raises before any byte exists   |
         v                                 v
   retry is SAFE:                   [mid-stream failure]
   nothing was rendered             keep buffer if partial text exists
                                    mark truncated, continue in a
                                    follow-up turn — never replay

Failures at open. Auth, payload, and rate-limit errors raise while the request is being established: AuthenticationError (401 — fix the key, never retry), BadRequestError (400 — your payload), RateLimitError (429 — back off), plus connect failures. The handler wraps the stream open, not the loop:

import httpx
from openai import OpenAI, APIConnectionError, APITimeoutError

client = OpenAI(
    timeout=httpx.Timeout(connect=5.0, read=60.0),  # read budget spans silence
    max_retries=0,                                   # recovery is yours on a stream
)

buf = []
try:
    for chunk in client.chat.completions.create(
        model="gpt-4o-mini", stream=True, messages=messages
    ):
        if chunk.choices:
            buf.append(chunk.choices[0].delta.content or "")
except (APIConnectionError, APITimeoutError):
    partial = "".join(buf)
    if not partial:                    # nothing rendered: full retry is safe
        retry_later(messages)
    else:                              # text already shown: never replay it
        record_truncated(partial)      # then continue in a follow-up turn

Input: a 400-token answer interrupted after Idempotency means retrying a.

Output: record_truncated stores the partial text, the UI marks the answer incomplete, and retry_later is skipped — re-emitting from the start would double text the user already read.

Two configuration details carry weight here. The read timeout spans the silence between frames, not the whole response, so a generous read budget with a tight connect budget is the correct asymmetry for streaming. And max_retries=0 disables the SDK's automatic replay: that helper re-sends the full prompt, and on a stream that died halfway, a replay is a duplicate answer, not a recovery.

Tool calls stream too. Argument payloads arrive as fragments — delta.tool_calls[i].function.arguments on OpenAI, input_json_delta.partial_json on Anthropic — and long JSON arguments are where the saved round trip pays off most: the fragments are already in memory while the model is still writing, ready to validate or dispatch the moment the stream signals completion.

args = ""
for chunk in stream:
    choice = chunk.choices[0]
    if choice.delta.tool_calls:
        args += choice.delta.tool_calls[0].function.arguments or ""
    if choice.finish_reason == "tool_calls":
        tool_call = json.loads(args)   # fragments form a complete object
        break

Input: a tool call streaming {"city": "Seattle", "unit": "celsius"} in fragments.

Output: the parsed dict the moment the closing brace arrives.

Parse only on the signal — finish_reason on OpenAI, stop_reason: tool_use on Anthropic. JSON is invalid until its last byte, so premature json.loads fails by design; what streaming buys is that the arguments are assembled and ready before the rest of the answer finishes. Backoff policy belongs to Rate Limiting and Retries, and the Anthropic event taxonomy is in Anthropic SDK Essentials.