Online Metrics for AI Features

The launch review shows a 90% thumbs-up rate on the new AI reply drafter. The room is pleased. Two months later, the support ticket volume is unchanged, and nobody can explain why the feature that everyone loves changes nothing.

The thumbs-up rate is a vanity metric: it measures whether users pressed a button, not whether the feature did anything for them. Users rate answers that feel pleasant, and a fluent, confident answer feels pleasant even when it fails to resolve the question. The signal is real but shallow — it says something about tone, nothing about outcomes.

Offline gates like the eval harness catch regressions before merge, but they run on a fixed dataset. Online metrics measure what happens when the feature meets the full chaos of production traffic: the questions nobody anticipated, the users who never give feedback, the cohorts the dataset underrepresented. The two layers are complementary — offline proves the change is safe, online proves it helps.

The metrics that prove a feature helps share one property: they are tied to an outcome the user or the business actually wants.

Online signals come in two kinds, and the distinction is trust versus volume.

Explicit signals are trustworthy when given but rare: most users never rate anything, and those who do are having a very good or very bad day. Implicit signals are behavioral traces — what users did, not what they said — and they are where the volume is. A follow-up rephrase ("no, I meant the second invoice") is a friction signal that never appears in a thumbs rating.

Instrumentation is the prerequisite: none of these metrics exist unless the events are emitted.

def track_event(session_id: str, event: str, props: dict) -> None:
    analytics.track(session_id, f"ai.{event}", {
        "session_id": session_id,
        "feature": props["feature"],
        "model": props["model"],
        "latency_ms": props["latency_ms"],
        # never the answer text or user identifiers here
    })

track_event(sid, "answer_shown", {...})
track_event(sid, "answer_copied", {...})
track_event(sid, "thumbs", {"value": 1})
track_event(sid, "task_completed", {"task": "refund_status"})

The comment in that function is the rule: events carry dimensions for slicing, never content. Answer text and personal identifiers belong in the trace store with access controls, not in the analytics pipeline.

The outcome metric for an AI feature is task completion: did the user finish the job the feature exists for? For a reply drafter, a completed task is a ticket resolved with the draft accepted. For a search assistant, it is the user finding the answer without opening a ticket. Completion ties the feature to a real outcome instead of engagement theater.

The supporting metrics are latency and cost, and both need percentiles, not averages:

import numpy as np

def session_metrics(events: list[dict]) -> dict:
    sessions = group_by_session(events)
    completed = [s for s in sessions if s.has("task_completed")]
    latencies = [s.latency_ms for s in sessions]
    return {
        "completion_rate": len(completed) / len(sessions),
        "p50_latency_ms": np.percentile(latencies, 50),
        "p95_latency_ms": np.percentile(latencies, 95),
        "cost_per_session": sum(s.cost_usd for s in sessions) / len(sessions),
    }

Input: 10,000 sessions in a week.

Output: completion_rate 0.61, p50 1.9s, p95 8.4s, cost_per_session $0.012.

The p95 line is the one that changes decisions. A mean latency of 2.4 seconds sounds fine; a p95 of 8.4 seconds says the slowest 5% of sessions — often the users on the weakest connections, with the longest inputs — are having a different product entirely. Percentiles expose the tail; averages launder it.

Cost per session is the unit economics: total model spend divided by sessions served. It is the number that answers "can we afford to make this feature better?" — a feature at $0.012 per session can absorb a bigger model for a quality gain; one at $0.40 cannot.

An overall completion rate of 61% is an average of unknown composition. It could be 61% everywhere, or 90% for English-speaking users on desktop and 30% for everyone else. The average cannot tell you, and the second world is a product with a broken experience hiding inside a fine dashboard.

Segmentation is the practice of slicing metrics by cohort before reading them:

The last row connects online metrics to the regression suite: a provider model update that degrades one cohort shows up here first, and the suite confirms it offline.

Failure modes of segmentation done badly:

• Segmenting after the complaint. Cohorts should be instrumented at launch, not after the first broken-slice ticket.
• Too many slices, no protected ones. Pick a small set of protected slices — paying customers, high-stakes intents — and gate releases on them explicitly.
• Reading only the aggregate. The aggregate is the last number you look at, not the first.
• Ignoring latency in segments. A cohort with fine completion and terrible p95 is a cohort about to churn.

Online metrics close the loop that offline evaluation opens. The harness proves a change is safe; these metrics prove it helped — for every cohort, not just the average.