Estimating Token Costs

Finance asks what the AI features will cost next quarter, and the honest answer today is a shrug: nobody can price a request yet. Pricing becomes arithmetic once the price card is read correctly, and price cards carry three quirks that break naive math.

An example card (real cards move; the structure does not):

Treat the table as a format rather than a fixture: providers revise prices often, but the input / cached-input / output structure is stable across them.

Three quirks, in the order they bite:

1. Rates are per million, not per token. A 1,800-token prompt is 0.0018M tokens; multiply by the card rate, never by 1,800.
2. Input and output bill separately. Output costs 4x input on this card — and output is the half you control least, since verbosity is a cost center.
3. A cached prefix bills at the cheaper rate. The system prompt and boilerplate you resend every call are eligible for the discount.

PRICE = {'in': 0.15, 'out': 0.60, 'cached': 0.075}   # USD per 1M tokens

def request_cost(input_tokens, output_tokens, cached_tokens=0):
    fresh = input_tokens - cached_tokens
    return (fresh * PRICE['in'] + cached_tokens * PRICE['cached']
            + output_tokens * PRICE['out']) / 1e6

Input: request_cost(1800, 150). Output: 0.00036 dollars.

The breakdown: 1,800 x 0.15 / 1e6 = 0.00027 for input, 150 x 0.60 / 1e6 = 0.00009 for output, total 0.00036. The /1e6 line is the one to explain aloud: it converts a per-million rate into a per-token rate before any multiplication, which is where unit mistakes hide.

A third of a cent per request looks like nothing. That is the trap — the number only becomes meaningful through the multiplicative stack behind it: tokens per request, requests per day, days per month, multiplied by features. The next chapter builds that stack per feature rather than for the product, because Context Window Budgeting already taught that input size is a design decision, not a constant.

One product-wide number hides the only useful question: which feature is expensive, and is it expensive because of input, output, or volume. Project per feature:

PRICE = {'in': 0.15, 'out': 0.60, 'cached': 0.075}   # USD per 1M tokens
DAYS = 30

def feature_month(name, in_tok, out_tok, calls_per_day, cached=0):
    per_call = ((in_tok - cached) * PRICE['in'] + cached * PRICE['cached']
                + out_tok * PRICE['out']) / 1e6
    per_day = per_call * calls_per_day
    return name, round(per_call, 6), round(per_day, 2), round(per_day * DAYS, 2)

for row in [
    feature_month('ticket-triage', 1800, 150, 50_000),
    feature_month('thread-summary', 6000, 700, 5_000),
    feature_month('triage-cached', 1800, 150, 50_000, cached=1200),
]:
    print(row)

Output:

('ticket-triage', 0.00036, 18.0, 540.0)
('thread-summary', 0.00132, 6.6, 198.0)
('triage-cached', 0.00027, 13.5, 405.0)

Read the three rows against each other. ticket-triage is input-heavy: 50,000 calls a day at 0.00036 each lands on $540 a month, and 75% of that is input — so trimming history or shrinking retrieval chunks moves the needle, while shortening the answer barely does. thread-summary is the opposite shape: 5,000 calls a day but 700 output tokens per call, so a third of its bill is the expensive output half.

The mix decides which lever pays: input is 75% of triage's per-call cost, while output is 32% of the summarizer's.

The third row is the discount. 1,200 of the 1,800 input tokens are a repeated prefix, billed at 0.075 instead of 0.15: per-call cost drops from 0.00036 to 0.00027, which is $4.50 a day and $135 a month saved for one caching rule on content you resend anyway. Cache-hit pricing only rewards prefixes that are literally identical, so the system prompt has to stop changing for the discount to apply.

Average token counts must come from logged traffic, not from encoding one hand-picked prompt: use observed input and output averages, and project with the output you actually generate rather than the max_tokens cap. Token Usage Metering is where those averages come from.

Projections are only as good as the measurements under them, so the loop closes with logging and alerts.

• Log usage per request with a feature tag. prompt_tokens and completion_tokens come back on every response; store them with tenant, feature, and trace id. Metering aggregates; Trace LLM Calls correlates.
• Alert on shape, not just total. Tokens per request for a feature should be stable. A jump from 1,800 to 7,000 means retrieval k changed, history eviction stopped running, or a loop started resending context — see Context Window Budgeting for the eviction that should have prevented it.
• Retries multiply input cost. Each attempt re-sends the full prompt, so three attempts on a 6,000-token prompt bill 18,000 input tokens for one logical call (Rate Limiting and Retries).
• Prompt edits are baseline changes. Adding 300 tokens to the system prompt at 50,000 calls/day is 15M input tokens a day: 15 x 0.15 / 1e6 = $2.25 a day, $67.50 a month, forever, from one edit that nobody reviewed as a cost change.

Averages come from the usage block of real responses — average prompt_tokens and completion_tokens over the last few thousand calls per feature, and recompute after every prompt change, because a projection built on pre-launch estimates decays within one release cycle.

Common estimation failures:

• Pricing one prompt by hand instead of taking p50 and p95 from logs — the tail is where overruns live.
• Ignoring cache discounts on system-heavy workloads, which materially overstates every projection.
• Projecting from max_tokens rather than observed output; caps bound the tail, they do not describe the average.
• A single product-wide figure — it cannot tell you which feature to cut when the budget tightens.

With per-feature numbers and alerts in place, spend becomes a dial: raise cache coverage, trim input, or cap output verbosity and the monthly figure moves predictably. When the input side is too fat to trim by hand, the next lever is sending less content at all — Strategies for Long Inputs — and setting the actual ceilings lives in Cost Budgeting and Limits.