A/B Testing AI Features

Prompt v2 is ready. The team believes it answers support questions better. The honest way to find out is an A/B experiment: some users get v1, some get v2, and you compare outcomes. The design decision that makes or breaks the experiment is randomization — and the rule is simple: randomize at the user level, never per request.

Per-request randomization is the classic mistake. If assignment happens per request, one user sees v1 in the morning and v2 in the afternoon. Their experience is contaminated, your two arms are no longer comparable, and the user gets an inconsistent product. User-level assignment means each user has one stable variant for the whole experiment.

import hashlib

def assign_variant(user_id: str, experiment: str) -> str:
    # hash user + experiment: deterministic (same user -> same bucket on
    # every request) and independent (different experiments -> uncorrelated
    # buckets, so concurrent experiments do not interfere)
    bucket = int(hashlib.sha256(
        f'{experiment}:{user_id}'.encode()
    ).hexdigest(), 16) % 100
    return 'treatment' if bucket < 50 else 'control'

assign_variant('u_123', 'prompt-v2') -> 'treatment'
assign_variant('u_123', 'prompt-v2') -> 'treatment'   # stable across requests
assign_variant('u_123', 'model-b')   -> 'control'     # independent experiment
assign_variant('u_456', 'prompt-v2') -> 'control'

The hash gives you three properties for free: stability (the same user always lands in the same bucket, so their experience is consistent), independence (a different experiment name produces uncorrelated assignments, so you can run experiments in parallel), and auditability (assignment is reproducible from the user id and experiment name — you can always answer 'which variant did this user see?').

Store the assignment with every event you log. The variant is not metadata; it is the join key for the entire analysis. Without it, you cannot compute anything.

An AI experiment lives or dies on its metrics — and the failure mode is not having too few, it is having the wrong ones. A prompt variant that raises helpfulness from 71% to 74% looks like a win until you notice refusals doubled: the model got 'helpful' by refusing more questions. The primary metric moved in the right direction and the product got worse.

The fix is guardrail metrics: metrics you do not want to move, tracked alongside the primary one. A win is only a win if the guardrails hold.

def evaluate(experiment: str):
    rows = query('''
        SELECT variant,
               AVG(helpful_rate)       AS primary,      -- pre-registered
               AVG(refusal_rate)       AS guardrail,    -- must not rise
               AVG(latency_p95_ms)     AS guardrail,    -- must not rise
               COUNT(DISTINCT user_id) AS users
        FROM experiment_events
        WHERE experiment = %s
        GROUP BY variant
    ''')
    # pre-registered before traffic flowed:
    #   primary = helpful_rate, minimum detectable effect = 2 points
    #   guardrails = refusal_rate, latency_p95
    #   minimum sample = 2,000 users per variant
    ...

Pre-registration is the discipline that makes the numbers mean anything. Before any traffic flows, write down: the primary metric, the minimum effect worth detecting, the guardrail metrics, and the sample size needed. Without this, you are free to try metrics until one shows a difference — with twenty metrics, one will cross p < 0.05 by chance alone, and you will ship a coin flip.

The sample size calculation deserves respect. Detecting a 2-point lift on a 71% baseline with 80% power needs roughly 2,000 users per variant; detecting a 0.5-point lift needs thirty-two times that. Teams that skip this either run experiments too short to conclude anything (and then argue from noise) or run them long enough to detect effects too small to matter. Both failure modes are expensive. The same rigor applies to building eval datasets offline — an experiment is just an eval set with a treatment arm.

The experiment ran for two weeks. Helpful rate is up 2.4 points, guardrails held, p = 0.03. Time to ship — except nobody checked what happens at 100% rollout, and the variant has a bug that only triggers on prompts longer than 4,000 tokens, which is 0.1% of traffic. At 5% rollout that bug hit fifty users. At 100% it hits fifty thousand.

Gradual rollout exists for exactly this: a harmful variant should hit a small slice of traffic before it hits everyone.

1%  ->  5%  ->  25%  ->  50%  ->  100%
 │      │       │       │        │
 each step: guardrails still holding? error rate? latency?
 kill switch: one flag returns 100% to control, instantly

The kill switch is non-negotiable. It is a configuration flag — not a deploy, not a code change — that returns all traffic to the control variant. When the 2 AM alert fires, the person on call needs to stop the bleeding in seconds, not wait for a rollback build. Feature-flag systems exist for this; if you do not have one, a single if statement reading from the job store you already run for queue-based processing is enough.

Each rollout step has an explicit gate: guardrails holding, error rate flat, latency within budget. If any gate fails, the rollout pauses — not rolls back, unless the harm is active. Pausing preserves the option to resume once the cause is understood; rolling back destroys the data you need to understand it.

The rollout schedule is also your blast-radius control. A variant that harms 0.1% of users harms 100 people at 1% rollout and 100,000 at 100%. Gradual rollout converts 'we shipped a bug to everyone' into 'we caught a bug at 5% and fifty users had a bad answer.'

A/B testing AI features has all the pitfalls of classical experimentation plus a set that are specific to probabilistic systems. The table below is the short list — the ones that have actually invalidated experiments.

The peeking row is the one that surprises people. In classical statistics, a p-value is valid only if you committed to a sample size in advance. Checking results every day and stopping at the first significance inflates the false-positive rate dramatically — with daily peeking over two weeks, a meaningless result crosses p < 0.05 more than half the time. Either pre-register the end date or use sequential testing methods designed for continuous monitoring.

The novelty effect is specific to AI products because the product is the model's behavior. Users notice when answers change — any change — and engage more for a few days. A prompt variant measured in its first week is measuring curiosity as much as quality. Let the experiment run past the novelty window before concluding, the same patience regression suites enforce for code changes.

When the experiment concludes, the decision framework is: primary metric up by more than the minimum detectable effect, guardrails held, sample size reached, effect stable past the novelty window. All four, or the variant does not ship. This is the same discipline as online eval metrics applied to live traffic — and the capstone production RAG assistant expects you to run it that way.