Caching Model Responses

Model calls are the most expensive operation in an AI backend: 1-15 seconds and real money per request. A surprising share of production traffic is the same question asked repeatedly — status checks, policy questions, 'what are your hours?' A cache turns those repeat questions into sub-millisecond lookups.

Exact-match caching is the simplest form: if the identical request was answered recently, return the stored answer. The subtle part is the cache key. Hashing only the prompt text is a bug waiting to happen:

import hashlib
import json
import time

TTL = 3600
cache = {}  # production: Redis with EXPIRE

def cache_key(request: dict, prompt_version: str) -> str:
    # canonical JSON: sorted keys so key order does not change the hash
    canonical = json.dumps(request, sort_keys=True)
    return hashlib.sha256(f'{prompt_version}:{canonical}'.encode()).hexdigest()

def complete(request: dict, prompt_version: str) -> str:
    key = cache_key(request, prompt_version)
    hit = cache.get(key)
    if hit and hit['expires'] > time.time():
        hit['hits'] += 1
        return hit['answer']  # skip the provider call entirely
    answer = call_model(request)
    cache[key] = {'answer': answer, 'expires': time.time() + TTL, 'hits': 1}
    return answer

complete({'model': 'gpt-4o-mini', 'messages': [...], 'temperature': 0}, 'v3')
--> miss: provider call, 1.2s, $0.0004
complete(same request, 'v3')
--> hit: 0.3ms, $0.0000
complete(same request, 'v4')        # prompt was edited
--> miss: different key, fresh generation

The key must cover everything that changes the answer: model, messages, temperature, tool definitions, and the prompt version. Miss any of them and you serve an answer computed under different assumptions — a correctness bug, not just a stale one. json.dumps(..., sort_keys=True) matters more than it looks: {'a':1,'b':2} and {'b':2,'a':1} are the same request, and without sorted keys they hash differently, silently halving your hit rate.

The economics are worth stating plainly. At 10,000 requests per day with a 40% hit rate, you skip 4,000 provider calls daily. At token costs of a fraction of a cent per call, that is the difference between a feature that pays for itself and one that bleeds.

Exact matching fails on the long tail: 'What is your return policy?' and 'How do I send something back?' are the same question with different words. Semantic caching answers with embeddings — store the question's vector, and on a new question, retrieve the nearest past question and reuse its answer if the similarity is high enough.

def semantic_complete(query: str, prompt_version: str) -> str:
    q_vec = embed(query)
    cached = vector_db.search(q_vec, top_k=1)
    if cached and cosine(q_vec, cached.vec) > 0.97:
        # near-identical question already answered — but only trust it if
        # it was generated under the current prompt version
        if cached.prompt_version == prompt_version:
            return cached.answer
    answer = call_model(query)
    vector_db.upsert(q_vec, answer, prompt_version)
    return answer

embed('What is your return policy?')   -> vec A
embed('How do I send something back?') -> vec B
cosine(A, B) = 0.981 > 0.97 -> cache hit, answer reused

The threshold is the whole game. At 0.90 you get a high hit rate and serve wrong answers to questions that merely look similar. At 0.99 you get almost no hits. Tune it on your own queries — this is a domain-specific knob, not a universal constant, and the same discipline applies as when evaluating embeddings.

The staleness risk is real and specific. A semantic hit means 'a similar question was answered before', not 'this answer is still true'. If the return policy changed last week, the cached answer is confidently wrong. Two mitigations: the prompt-version gate in the code above, and a shorter TTL for semantic entries than exact ones. Semantic caching is a cost optimization with an error budget — use it for traffic where a slightly stale answer is acceptable, and keep exact matching for anything factual.

You ship a prompt edit on Friday. On Saturday, users report the old wording in answers. The cache is serving pre-edit responses, and nobody included the prompt version in the key. This is the most common caching bug in AI systems, and it is entirely preventable.

Invalidation in an AI cache works on two axes: identity and time.

Identity means the key must contain everything that defines the answer — including the prompt version, as in the code above. When the prompt changes, the key changes, old entries become unreachable garbage, and they expire on their own. No flush endpoint, no cache-wide invalidation job, no race between deploy and purge.

Time means the TTL must match how fast the underlying truth changes:

The live-data row is the one that bites. Caching a stock answer for 60 seconds is a defensible trade; caching it for an hour is a support ticket. When the answer depends on state you do not control, either skip the cache or make the TTL shorter than the state's change interval.

There is a subtler failure: the version gate only works if the version actually changes. If your prompt pipeline builds the system prompt from ten components, the cache key must hash the assembled prompt or a version of the whole assembly — a version bump to one component that does not reach the key leaves stale answers in place.

Caching an AI backend introduces failure modes that have nothing to do with the model. The ones below are the ones that actually reach production.

The thundering herd deserves the most engineering attention. When a popular entry expires, every concurrent request misses simultaneously and they all dispatch provider calls — the cache amplifies the load spike it was built to absorb. The fix is request coalescing: the first miss for a key becomes the caller, and concurrent requests for the same key wait on that call's result instead of starting their own. Memcached and Redis both have primitives for this (SET NX as a lock), and it is twenty lines of code.

The poisoning row is specific to AI systems. A cached answer is content you will show users without a model in the loop. If one user's prompt injection gets cached under a semantic or exact key, every subsequent match receives it. Validate and sanitize before storing, treat the cache as an output surface, and remember that caching interacts with cost budgeting: a high hit rate is a cost win, but only if the hits are correct. Measure hit rate and answer quality together, the same way you would for online eval metrics.