Queue-Based AI Processing

Your generation endpoint works fine in staging. Then a marketing email goes out and 4,000 users hit it in ten minutes. Each generation takes 8 seconds. Your web tier holds 200 concurrent requests. The math is brutal: 4,000 requests x 8 seconds / 200 workers = 160 seconds of queued waiting, and most HTTP clients time out at 30. The endpoint does not get slower — it stops accepting work at all.

This is the fundamental problem with synchronous generation: the HTTP request's lifetime is coupled to the model's runtime. Any spike in traffic becomes a spike in held connections, and held connections are the first resource a server exhausts.

The fix is to decouple accepting work from doing it:

Synchronous (collapses under spikes):
Client │── POST /generate ──▶ Server ── model call (8s) ──▶ response
       └─ connection held for the full 8s; 200 concurrent = hard ceiling

Queue-based (absorbs spikes):
Client │── POST /jobs ──▶ Server ── enqueue ──▶ 202 Accepted (ms)
Client │── GET /jobs/{id} ◀── status ◀── Queue ◀── Workers (scale independently)
       └─ HTTP tier stays fast; workers drain at their own rate

The queue is a buffer with one job: absorb the spike so the HTTP tier never holds a connection for the duration of a generation. Workers pull jobs at whatever rate the providers allow and scale independently of the web tier. This is the same pattern as async HTTP clients applied to the server side — never block the accepting layer on the slow layer. The rest of this topic builds the pieces: the submission flow, idempotency, retries, and the dead-letter queue.

A queue-based generation pipeline has four parts: an acceptance endpoint, a queue, workers, and a status endpoint.

                    ┌─────────────┐
   POST /jobs ─────▶│  HTTP tier  │──── enqueue ────┐
                    └─────────────┘                 │
                                                    ▼
                                              ┌──────────┐
                    ┌──────────┐  pull        │  Queue   │
                    │ Worker 1 │◀─────────────│ (Redis / │
                    └──────────┘              │  SQS /   │
                    ┌──────────┐              │  Rabbit) │
                    │ Worker 2 │◀─────────────│          │
                    └──────────┘              └──────────┘
                                                    │ done
                                                    ▼
   GET /jobs/{id} ◀── status ◀──────────────┌─────────────┐
                                            │  Job store  │
                                            └─────────────┘

The acceptance endpoint does three things: validate the request, enqueue a job, and return 202 Accepted with a job id. It never calls a model. The status endpoint is what the client polls — every 2 seconds is typical — until the job reaches a terminal state.

The queue itself can be anything durable: Redis lists for a single service, SQS or RabbitMQ when you want managed retries and visibility. The workers are the only components that touch providers, which means provider rate limits and outages affect workers, not users. When traffic doubles, you add workers; when it halves, you remove them. The HTTP tier never changes.

Job states follow a small state machine: queued -> running -> done, with failed and dead as terminal states for jobs that exhaust retries. Keep the state machine small — every state you add is a state your status endpoint, your monitoring, and your on-call runbook all have to understand.

Two properties separate a queue that survives production from one that survives a demo: idempotency and bounded retries.

Idempotency keys. Clients retry. Networks drop responses. Without idempotency, a retried submission runs the generation twice — and you pay twice. The fix is a key derived from the request itself:

import hashlib
import redis

r = redis.Redis()

def submit_job(user_id: str, prompt: str) -> str:
    # the key is a hash of the request identity: the same user submitting
    # the same prompt twice maps to the same key
    idem_key = hashlib.sha256(f'{user_id}:{prompt}'.encode()).hexdigest()[:16]
    existing = r.get(f'idem:{idem_key}')
    if existing:
        return existing.decode()  # retry: return the original job, do not re-enqueue
    job_id = f'job_{idem_key}'
    r.set(f'idem:{idem_key}', job_id, ex=3600)
    r.hset(f'job:{job_id}', mapping={'prompt': prompt, 'status': 'queued', 'attempts': 0})
    r.rpush('generation:queue', job_id)
    return job_id

POST /jobs {'user_id': 'u_123', 'prompt': 'Summarize Q3'}
--> 202 {'job_id': 'job_9f2c...'}

POST /jobs {'user_id': 'u_123', 'prompt': 'Summarize Q3'}   (client retry)
--> 202 {'job_id': 'job_9f2c...'}   (same job, no second generation)

Bounded retries. Workers retry transient failures — a 429, a 500, a dropped connection — but a job that fails for a structural reason (malformed prompt, unsupported language, a bug in your request builder) will fail forever. Cap the attempts and move the job aside:

MAX_ATTEMPTS = 3

def worker():
    while True:
        job_id = r.blpop('generation:queue', timeout=5)[1].decode()
        attempts = int(r.hget(f'job:{job_id}', 'attempts') or 0)
        if attempts >= MAX_ATTEMPTS:
            r.rpush('generation:dlq', job_id)   # dead-letter: stop retrying
            r.hset(f'job:{job_id}', 'status', 'dead')
            continue
        try:
            prompt = r.hget(f'job:{job_id}', 'prompt').decode()
            answer = call_model(prompt)          # the expensive part
            r.hset(f'job:{job_id}', mapping={'status': 'done', 'answer': answer})
        except TransientError:
            r.hincrby(f'job:{job_id}', 'attempts')
            r.rpush('generation:queue', job_id)  # re-enqueue for a later attempt

The status endpoint is deliberately boring:

@app.get('/jobs/{job_id}')
def job_status(job_id: str):
    status = r.hget(f'job:{job_id}', 'status')
    if status == b'done':
        return {'status': 'done', 'answer': r.hget(f'job:{job_id}', 'answer').decode()}
    return {'status': (status or b'unknown').decode()}  # client polls every 2s

GET /jobs/job_9f2c...  (every 2s)
--> {'status': 'queued'} --> {'status': 'running'} --> {'status': 'done', 'answer': '...'}

Polling every 2 seconds for a 10-second job means five requests per generation — cheap, and it keeps the client in control. Webhooks are the alternative when you cannot tolerate the poll latency, but they add delivery retries and signature verification, so start with polling.

The dead-letter queue (DLQ) is where jobs go when they have failed enough times that retrying is waste. The name sounds dramatic; the practice is mundane and essential. Without a DLQ, poison jobs sit at the head of the queue, fail, get retried, fail again — consuming worker capacity and provider quota forever while real jobs wait behind them.

A DLQ is a second queue plus two operational habits:

def replay_dlq():
    # run after fixing the root cause: re-queue dead jobs with attempts reset
    while True:
        job_id = r.blpop('generation:dlq', timeout=5)[1].decode()
        r.hset(f'job:{job_id}', mapping={'status': 'queued', 'attempts': 0})
        r.rpush('generation:queue', job_id)

Habit one: inspect before replay. Every dead job carries a failure reason. Group them — usually three or four causes account for nearly all of them — fix the cause, then replay. Blind replay just re-creates the DLQ.

Habit two: alert on DLQ depth. A growing DLQ is a production incident in progress, not a backlog. Alert when depth crosses a threshold so a poison job is caught in minutes, not discovered in a weekly review.

Failure modes specific to queue-based generation:

The worker-crash row is the one people miss. A worker that dies after pulling a job but before marking it done leaves the job in running limbo. Every production queue has a visibility timeout for exactly this: if a job is not acknowledged within the window, it returns to the queue. If you build on Redis lists without a queue library, you must implement this yourself — which is the strongest argument for using SQS, RabbitMQ, or Celery rather than rolling your own.

Queues turn your slowest, most expensive operation into background work your HTTP tier never waits on. The cost is operational: you now run workers, a queue, and a job store, and you need token usage metering to know what the background work is spending. When generation is too slow even for a queue — real-time chat, interactive assistance — that is what streaming APIs are for.