Streaming APIs for AI Backends

A user submits a prompt and watches a spinner. Twelve seconds later, the full answer appears in one block. The answer is correct, but the user has already tabbed away. This is the failure mode streaming exists to fix: perceived latency is governed by time-to-first-token (TTFT), not total generation time.

A typical chat completion takes 5-15 seconds end to end, but the first token usually arrives in under a second. Streaming sends each token to the client the moment the provider emits it, so the user starts reading while the model is still generating. The difference between a 12-second wait and a 1-second wait is the difference between a feature that feels broken and one that feels instant.

Non-streaming request:
Client │──── POST /chat ──────────▶ Server │──── provider call (12s) ────▶
Client │◀──── full answer ─────────────── Server
       └─ user sees nothing for 12s

Streaming request:
Client │──── POST /chat/stream ─────▶ Server │── provider stream ──▶
Client │◀── tok ── tok ── tok ── tok ── [DONE]
       └─ first token in ~800ms, user reads while the rest generates

The two flows cost the same in tokens and dollars. Streaming only changes when bytes move. That is why every major provider exposes a stream: true mode, and why streaming responses are the default shape of LLM product UX. The rest of this topic covers the server side: building the endpoint, keeping it alive through production infrastructure, and stopping payment for tokens nobody reads.

Server-sent events (SSE) is a one-way push protocol over plain HTTP. The server responds with Content-Type: text/event-stream and writes a series of events, each formatted as data: <payload>\n\n. The browser's EventSource API consumes them and auto-reconnects if the socket drops.

SSE fits token streams better than a raw WebSocket upgrade for three reasons:

• One-way is all you need. Tokens flow server to client; the client never sends mid-stream messages.
• It survives proxies. SSE is an ordinary HTTP response body, so it passes through load balancers and reverse proxies that do not support WebSocket upgrades.
• Reconnection is built in. EventSource handles retries with backoff automatically; a hand-rolled WebSocket client must implement that itself.

Here is a minimal FastAPI endpoint that relays a provider's token stream:

import json
import httpx
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()
PROVIDER_URL = 'https://api.openai.com/v1/chat/completions'

@app.post('/chat/stream')
async def chat_stream(request: Request):
    body = await request.json()

    async def generate():
        async with httpx.AsyncClient(timeout=60.0) as client:
            async with client.stream('POST', PROVIDER_URL, json=body) as resp:
                async for line in resp.aiter_lines():
                    if not line.startswith('data:'):
                        continue
                    payload = line[6:].strip()
                    if payload == '[DONE]':
                        break
                    token = json.loads(payload)['choices'][0]['delta'].get('content', '')
                    if token:
                        yield 'data: ' + json.dumps({'token': token}) + '\n\n'
            yield 'data: [DONE]\n\n'

    return StreamingResponse(
        generate(),
        media_type='text/event-stream',
        headers={'Cache-Control': 'no-cache', 'X-Accel-Buffering': 'no'},
    )

Request and response:

POST /chat/stream
{'model': 'gpt-4o-mini', 'messages': [{'role': 'user', 'content': 'Explain SSE'}], 'stream': true}

<-- 200 OK (text/event-stream)
data: {'token': 'Server'}

data: {'token': '-sent'}

data: {'token': ' events'}

data: [DONE]

Two details carry real weight. X-Accel-Buffering: 'no' tells nginx not to buffer the response — without it, nginx accumulates chunks and the stream arrives in bursts, defeating the purpose. Cache-Control: 'no-cache' stops intermediaries from caching a response that is still being generated. The aiter_lines loop is the backpressure boundary: the provider's bytes only move when the client reads them, which is what makes disconnect detection in the next chapter possible.

The relay loop above has two production bugs. First, if the user closes the tab, the loop keeps reading from the provider, and the provider keeps billing tokens for a client that no longer exists. Second, if the provider pauses between tokens for longer than the load balancer's idle timeout, the balancer reaps the connection and the user's stream dies mid-sentence.

Both fixes live in the same loop:

import time

KEEPALIVE_INTERVAL = 15  # seconds

async def generate():
    last_activity = time.monotonic()
    async with httpx.AsyncClient(timeout=60.0) as client:
        async with client.stream('POST', PROVIDER_URL, json=body) as resp:
            async for line in resp.aiter_lines():
                # Starlette exposes the socket state; when the client hangs up
                # this returns True and we unwind immediately
                if await request.is_disconnected():
                    await resp.aclose()  # stop the provider stream: no more tokens, no more bill
                    return
                if not line.startswith('data:'):
                    continue
                payload = line[6:].strip()
                if payload == '[DONE]':
                    break
                token = json.loads(payload)['choices'][0]['delta'].get('content', '')
                if token:
                    last_activity = time.monotonic()
                    yield 'data: ' + json.dumps({'token': token}) + '\n\n'
                elif time.monotonic() - last_activity > KEEPALIVE_INTERVAL:
                    # lines starting with ':' are comments — EventSource ignores
                    # them, but the bytes keep the socket warm through proxies
                    yield ': keep-alive\n\n'
                    last_activity = time.monotonic()
    yield 'data: [DONE]\n\n'

request.is_disconnected() is the check that matters most for cost. A single abandoned generation is trivial; a thousand abandoned tabs running to completion is a four-figure monthly surprise. Cancelling the upstream stream the moment the client leaves converts wasted spend into a sub-second truncation.

The keep-alive comment solves the opposite problem: a stream that goes silent. Many load balancers and cloud proxies kill idle TCP connections after 30-60 seconds, and a long generation with a thinking pause between tokens can look idle and get reaped. Emitting : keep-alive every 15 seconds produces traffic the client ignores but the infrastructure sees.

Buffering is the third lever. Yielding per token is correct but chatty on slow networks; some servers batch tokens into 100 ms windows before flushing. The tradeoff is latency versus packet efficiency — for interactive chat, flush per token; for bulk transcription relay, batching is fine.

Streaming endpoints fail in ways synchronous endpoints do not. The failure modes below are the ones that show up in production incident reviews, roughly in order of how often they bite.

Two of these deserve emphasis. The provider stall is the nastiest: aiter_lines blocks forever if the provider accepts the request and then never sends bytes. Always set an explicit httpx.Timeout on the upstream client — the default is no timeout at all. The reconnect problem is subtle: EventSource retries automatically, so a dropped connection means the client re-requests and the server starts a second generation for a user who already has the answer. Send a terminal data: [DONE] event and treat a reconnect carrying a session id as a resume, not a new generation.

Streaming is the default shape of LLM UX, which puts the endpoint on your critical path. Treat it like one: load test it with slow clients, alert on upstream truncation rates, and read async HTTP patterns before wiring the client. When the work cannot stream — long reports, batch summarization — the answer is queue-based processing, the next topic in this phase.