Budgeting AI Spend

Your monthly AI bill is $40,000. A quarter of it is classification calls — 'is this ticket billing, technical, or account?' — running on your most expensive frontier model. That is not a model problem; it is a routing problem.

The fix is model tiers. Not every task needs a frontier model:

The routing decision is a policy, not a per-call guess. Classify the task at the edge — by feature flag, by tool schema, by a cheap classifier — and route to the tier it needs. The foundation model landscape is where you pick the specific models for each tier.

The savings are not hypothetical. Routing trivial tasks to a small model typically cuts 30–60% of total spend with no measurable quality change, because the tasks were never hard. The frontier model is reserved for the calls where its reasoning actually moves the needle — which is also where its latency is justified.

Prompt caching is the cheapest optimization in this topic because it does not change behavior — it changes billing. Providers discount tokens that arrive as a cache hit, often by 50–90%, and the discount applies to the stable prefix of your prompt.

The mechanism: when a request's prompt prefix matches a previously processed prefix, the provider skips prefill for those tokens and bills them at the cache rate. The prefix must match exactly — token-for-token — which means the stable part must come first.

# BAD: user input first, system prompt second — the prefix changes every request
messages = [
    {'role': 'user', 'content': user_question},
    {'role': 'system', 'content': SYSTEM_PROMPT},  # never a cache hit
]

# GOOD: stable system prompt first, variable input last
messages = [
    {'role': 'system', 'content': SYSTEM_PROMPT},  # cached across requests
    {'role': 'user', 'content': user_question},    # only this part is new
]

The ordering is the whole trick. A 2,000-token system prompt that sits in front of every request becomes a cache hit for every request; the same prompt placed after the user input is a cache miss every time.

Caching compounds with response caching for repeated queries, but prompt caching is the larger win for system-heavy applications — RAG pipelines, agents with long system prompts, and any feature where the prompt is mostly static. Estimate the effect with token cost estimation: multiply cached tokens by the cache-hit discount and subtract from the monthly projection.

Budgets that live in a spreadsheet are aspirations. Budgets that live in the request path are controls. The enforcer sits between your application and the model provider and makes a binary decision: allow or reject.

from dataclasses import dataclass

@dataclass
class Budget:
    feature: str
    monthly_cap_usd: float
    alert_at: float = 0.8

class BudgetEnforcer:
    def __init__(self, spend_store):
        self.spend = spend_store  # feature -> USD spent this month

    def check(self, feature: str, budget: Budget, estimated_cost_usd: float) -> dict:
        spent = self.spend.get(feature, 0.0)
        projected = spent + estimated_cost_usd
        if projected > budget.monthly_cap_usd:
            return {'allow': False, 'reason': 'budget_exceeded'}
        if projected >= budget.monthly_cap_usd * budget.alert_at:
            return {'allow': True, 'alert': 'budget_threshold'}
        return {'allow': True}

enforcer = BudgetEnforcer(spend_store={'support-chat': 9200.0})
budget = Budget(feature='support-chat', monthly_cap_usd=10000.0)

print(enforcer.check('support-chat', budget, estimated_cost_usd=500.0))
print(enforcer.check('support-chat', budget, estimated_cost_usd=1000.0))

Output:

{'allow': True, 'alert': 'budget_threshold'}
{'allow': False, 'reason': 'budget_exceeded'}

The non-obvious line is the alert check using projected, not spent. Alerting on current spend pages you once and goes quiet; alerting on the projected spend — current plus this request — catches the threshold crossing as it happens, while the request is still small enough to matter.

Set budgets per environment, not just per feature. A production cap of $10,000 and a staging cap of $50 means a runaway test loop costs $50, not $10,000. The kill switch is the same code path with a lower number — which is why it is a control and not a policy.

Cost per call is the metric that feels precise and misleads. A feature that makes 10 calls at $0.01 each looks cheaper than one that makes 1 call at $0.05 — until you learn the 10-call feature succeeds 40% of the time and the 1-call feature succeeds 95%.

The metric that survives contact with reality is cost per successful task:

def cost_per_success(total_spend: float, attempts: int, successes: int) -> float:
    if successes == 0:
        return float('inf')
    return total_spend / successes

# Feature A: 10 calls, 4 successes, $0.40 total
print(cost_per_success(0.40, 10, 4))   # 0.100

# Feature B: 1 call, 1 success, $0.05 total
print(cost_per_success(0.05, 1, 1))    # 0.050

Feature B is half the cost per success despite a higher per-call price. The per-call metric ranked them backwards.

This is why quality improvements show up as real savings. A better prompt that raises success rate from 40% to 80% halves the cost per success without touching the model, the prompt length, or the cache. A reranker that reduces retries from 3 to 1 does the same. The savings are invisible if you only watch per-call cost.

Track it per feature, per day, alongside the per-feature caps from metering token usage. When cost per success rises, the cause is usually a quality regression — which is an incident response problem, not a budget problem.