Metering Token Usage

The provider invoice arrives: $12,400 this month, up 340%. You know the total. You do not know which feature caused it, which tenant to bill, or whether the spike is a runaway loop or a legitimate launch. Token usage without dimensions is a number you cannot act on.

Every request that reaches a model must be tagged before it is sent:

@dataclass
class UsageEvent:
    feature: str        # 'support-chat', 'doc-extraction'
    tenant: str         # 'acme-corp'
    environment: str    # 'production', 'staging'
    model: str
    input_tokens: int
    output_tokens: int
    timestamp: str

These three dimensions — feature, tenant, environment — turn a single total into answers. Which feature is growing? Which tenant is over their plan? Is staging burning production-rate tokens on a test loop?

The environment tag is the one teams skip and regret. A staging pipeline that calls the model in a loop generates real tokens at real prices. Without the environment dimension, that spend hides inside the production total and surfaces only as a line item you cannot explain.

Tagging is also the foundation for everything downstream: per-customer alerts, chargeback, and the dashboards that catch a runaway agent loop in hours instead of weeks. If you have not read how tokenizers work, the token counts here are the same counts the provider bills on — which is why metering must use the provider's usage field from the response, not your own estimate.

Raw events are a firehose. Dashboards need aggregates. The aggregation step groups usage events by dimension and time bucket so a dashboard can answer 'how many tokens did support-chat burn today?' without scanning every request.

from collections import defaultdict
from datetime import datetime, timedelta

def aggregate_daily(events, day):
    '''Sum tokens per feature for one UTC day.'''
    totals = defaultdict(lambda: {'input': 0, 'output': 0, 'requests': 0})
    day_start = datetime.fromisoformat(day)
    day_end = day_start + timedelta(days=1)
    for e in events:
        ts = datetime.fromisoformat(e['timestamp'])
        if day_start <= ts < day_end:
            key = (e['feature'], e['tenant'])
            totals[key]['input'] += e['input_tokens']
            totals[key]['output'] += e['output_tokens']
            totals[key]['requests'] += 1
    return dict(totals)

events = [
    {'feature': 'support-chat', 'tenant': 'acme', 'environment': 'production',
     'model': 'claude-sonnet-4-6', 'input_tokens': 1200, 'output_tokens': 300,
     'timestamp': '2026-09-30T10:15:00'},
    {'feature': 'support-chat', 'tenant': 'acme', 'environment': 'production',
     'model': 'claude-sonnet-4-6', 'input_tokens': 900, 'output_tokens': 250,
     'timestamp': '2026-09-30T10:16:00'},
    {'feature': 'doc-extraction', 'tenant': 'acme', 'environment': 'staging',
     'model': 'claude-haiku-4-5', 'input_tokens': 4000, 'output_tokens': 800,
     'timestamp': '2026-09-30T10:16:30'},
]

print(aggregate_daily(events, '2026-09-30'))

Output:

{('support-chat', 'acme'): {'input': 2100, 'output': 550, 'requests': 2},
 ('doc-extraction', 'acme'): {'input': 4000, 'output': 800, 'requests': 1}}

The non-obvious line is the defaultdict(lambda: ...). A plain dict would raise KeyError on the first unseen feature-tenant pair; the defaultdict creates the zeroed bucket on first access, so the aggregation loop stays a single pass with no existence checks.

Group by (feature, tenant) and the dashboard question answers itself: support-chat burned 2,650 tokens across two requests, doc-extraction burned 4,800 in one. The staging environment tag on the second row is what tells you that 4,800-token request is a test loop, not customer traffic.

In production this aggregation runs as a streaming job over the event log, writing rollups to a time-series store. The logic is identical; only the execution model changes.

Aggregates are history. Alerts are the present. The goal is to know about a runaway loop before the invoice does — while there is still time to kill the deploy.

The simplest effective alert is a per-customer spend threshold. Each tenant has a plan limit; when their trailing 24-hour spend crosses 80% of it, page the team. At 100%, the proxy rejects new requests with a clear error rather than letting the tenant blow through the month.

def check_spend_alert(tenant_spend_24h: float, plan_limit: float) -> str | None:
    ratio = tenant_spend_24h / plan_limit
    if ratio >= 1.0:
        return 'BLOCK'
    if ratio >= 0.8:
        return 'PAGE'
    return None

Thresholds catch known failure modes: a buggy integration retrying in a loop, an agent that never terminates, a customer scraping your API. They do not catch slow leaks — a feature that quietly doubles its token use over a week. For that, compare each day's per-feature total against a rolling baseline:

def detect_spike(today: int, baseline_avg: float) -> bool:
    return baseline_avg > 0 and today > baseline_avg * 2.5

A 2.5x jump over the trailing 14-day average is the signal that something changed — a new prompt that doubled context, a retrieval config that started pulling ten chunks instead of three, a model swap with different tokenization. The baseline matters: alerting on raw totals pages you every Monday morning.

Route these alerts to the team that owns the feature, not a generic ops channel. The support-chat on-call can judge whether a spike is a launch or a loop; a central team cannot.

Your metering says you spent $11,200. The provider invoice says $12,400. The $1,200 gap is drift, and it is normal. The question is whether your drift is 2% or 20%.

Drift has a short list of causes:

Reconcile monthly, mechanically:

1. Pull the provider invoice's per-model token totals.
2. Pull your metering's per-model token totals for the same window.
3. Compute the ratio per model. A ratio within 1.0–1.1 is healthy; above 1.2 means your metering is missing something the provider is billing.
4. Investigate the largest gap first — it is usually one cause, not five.

Reconciliation is also your defense against the 'our dashboards are wrong' conversation. When finance asks why the bill is $12,400, the answer is not 'the meter is approximate' — it is 'the meter tracks within 3% of the invoice, and here is the breakdown.' That answer comes from tracing LLM calls for the request-level detail and metering for the aggregate view.