Experiment Design

Why Experiment Design Matters

Most A/B test failures are not statistical failures. They are design failures. An experiment that is poorly designed cannot be rescued by sophisticated analysis. If the treatment and control groups differ systematically before the treatment is applied, no amount of post-hoc adjustment will produce a valid causal estimate.

Consider a team that tests a new checkout flow. They assign users to treatment based on whether their user ID is even or odd. The treatment group gets the new flow. After two weeks, the treatment group shows a 3% lift in conversion. The team ships it. Six weeks later, conversion drops below baseline. What went wrong? The even/odd split correlated with account creation date — older accounts had lower IDs and different purchasing behavior. The groups were never comparable.

This is the central problem experiment design solves: ensuring that the only systematic difference between groups is the treatment itself. Every design decision — how you randomize, what you randomize over, how you verify balance — serves this single goal.

The Three Pillars of Experiment Design

1. Randomization: Assigning users to groups using a process that is independent of user characteristics. This ensures groups are comparable in expectation, both on observed and unobserved variables.

2. Unit of randomization: The entity that gets assigned to a group. This must match the unit at which the treatment is applied. Randomizing by user but applying treatment by session creates contamination.

3. Pre-registration: Documenting the hypothesis, primary metric, sample size, and analysis plan before looking at the outcome data. This prevents the researcher from trying multiple analyses and reporting only the significant one.

What Design Errors Cost You

Design errors are expensive because they are silent. A biased experiment produces a confident, precise, wrong answer. You do not get a warning. You get a number that looks exactly like a valid result.

• Selection bias: Groups differ at baseline. The estimated treatment effect is confounded with pre-existing differences.
• Contamination: Control group members receive the treatment (or vice versa). The estimated effect is biased toward zero.
• Peeking without correction: Checking results repeatedly and stopping when significant. False positive rate inflates far beyond the nominal 5%.
• Sample ratio mismatch: The observed ratio of treatment to control users differs from the intended ratio. This signals a bug in assignment, logging, or data collection.

Each of these has a design-level fix. None of them can be fixed after the data is collected. This is why design is the highest-leverage phase of any experiment.

Randomization and the Unit of Randomization

Randomization is the mechanism that makes causal inference possible. Without it, you are running an observational study with extra steps.

What Randomization Actually Does

Randomization does not guarantee that treatment and control groups are identical. It guarantees that they are comparable in expectation. Any difference between groups before treatment is due to chance, not systematic bias. This is what allows you to attribute post-treatment differences to the treatment itself.

Formally, randomization ensures that the assignment mechanism is independent of potential outcomes. If Y(1) and Y(0) are the outcomes under treatment and control, randomization ensures that assignment A is independent of (Y(1), Y(0)). This is the foundation of the Rubin causal model.

Choosing the Unit of Randomization

The unit of randomization is the entity that gets assigned to a group. The most common units are:

The critical rule: the unit of randomization must match the unit at which the treatment is applied. If you randomize by user but the treatment only affects sessions where the user is logged in, you have a mismatch. Some sessions in the treatment group will not receive the treatment, diluting your effect.

User-Level Randomization

Most experiments randomize by user. This is the default choice because most treatments are user-specific. The user is assigned to a group when they first encounter the experiment, and they remain in that group for the duration of the experiment.

import hashlib

def assign_group(user_id: str, experiment_id: str, salt: str = "exp_2024_q3") -> str:
    key = f"{experiment_id}:{salt}:{user_id}"
    hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16)
    return "treatment" if hash_val % 2 == 0 else "control"

This approach is deterministic — the same user always gets the same assignment. This is important for consistency. A user who sees the treatment on Monday should not see the control on Tuesday.

Session-Level Randomization

Session-level randomization is appropriate when the treatment is not tied to user identity. A search algorithm experiment might randomize by session because the same user may have different search intents on different visits.

The risk with session-level randomization is carryover effects. A user who had a bad experience in a treatment session may behave differently in their subsequent control session. This violates the independence assumption.

Cluster Randomization

Sometimes you must randomize by group rather than by individual. If the treatment is a new pricing page, you cannot show different prices to users in the same region — it would be unfair and possibly illegal. You randomize by region instead.

Cluster randomization reduces effective sample size. Users within a cluster are correlated, so 1000 users in 10 clusters provide less information than 1000 users in 1000 clusters. You must account for this in your sample size calculation using the design effect.

Practical Assignment Strategies

• Simple randomization: Each user has a fixed probability of assignment. Works well for large samples.
• Blocked randomization: Assignments are made in blocks to ensure balance over time. Prevents a situation where the first week of an experiment is 70% treatment.
• Stratified randomization: Randomize within strata (e.g., platform, geography) to ensure balance on important covariates. Useful when sample size is small.

Sample Ratio Mismatch

Sample ratio mismatch (SRM) occurs when the observed ratio of users in treatment to control differs significantly from the intended ratio. If you designed a 50/50 split but observe 52/48, something is wrong.

SRM is one of the most reliable indicators of a broken experiment. It is a canary in the coal mine. When you detect SRM, you should stop the experiment and investigate before looking at any outcome metrics.

Why SRM Happens

SRM is always caused by a bug or a systematic difference in data collection. Common causes:

1. Logging bugs: The treatment group logs events differently from the control group. Maybe the treatment page has a JavaScript error that prevents the tracking pixel from firing.

2. Bot filtering: Bots are filtered from one group but not the other. If the treatment page loads differently, it may attract different bot traffic.

3. Caching issues: A CDN caches the treatment page but not the control page, causing some users to be served the wrong version.

4. Assignment timing: Users are assigned before a key event (e.g., login) but the event only fires for some users. If treatment users are more likely to log in, the post-login ratio will be skewed.

5. Data pipeline delays: One group's data arrives with a delay, creating a temporary mismatch that resolves itself.

Detecting SRM

SRM is detected with a chi-squared test comparing observed counts to expected counts.

import numpy as np
from scipy.stats import chi2

def check_srm(observed_treatment: int, observed_control: int, expected_ratio: float = 0.5) -> dict:
    total = observed_treatment + observed_control
    expected_treatment = total * expected_ratio
    expected_control = total * (1 - expected_ratio)
    
    chi2_stat = ((observed_treatment - expected_treatment) ** 2 / expected_treatment +
                  (observed_control - expected_control) ** 2 / expected_control)
    p_value = 1 - chi2.cdf(chi2_stat, df=1)
    
    return {
        "chi2_stat": chi2_stat,
        "p_value": p_value,
        "observed_ratio": observed_treatment / total,
        "expected_ratio": expected_ratio,
        "has_srm": p_value < 0.001
    }

result = check_srm(observed_treatment=52000, observed_control=48000)
print(f"SRM detected: {result['has_srm']}")
print(f"p-value: {result['p_value']:.2e}")

Use a strict threshold (p < 0.001) for SRM detection. A standard 0.05 threshold will produce false alarms because you are running this check on every experiment. The 0.001 threshold accounts for this multiple comparisons problem.

What to Do When You Detect SRM

1. Do not look at outcome metrics. Any result you see is uninterpretable.
2. Check the assignment mechanism. Verify that the randomization code is working correctly.
3. Check data collection. Compare event volumes between groups for events that should be identical (e.g., page views).
4. Check for differential attrition. Are users in one group more likely to drop out before the experiment ends?
5. Segment the data. Is the mismatch concentrated in a specific platform, geography, or time period?

Preventing SRM

• Run a AA test before the real experiment. Assign users to two control groups and verify that no metric differs significantly. This validates your assignment and logging infrastructure.
• Monitor SRM during the experiment, not just at the end. Set up automated alerts.
• Log assignment events separately from outcome events. This allows you to distinguish between assignment bugs and logging bugs.

Pre-registration and Hypothesis Documentation

Pre-registration is the practice of documenting your experiment plan before you look at the outcome data. It is the single most effective defense against p-hacking and publication bias.

The Problem Pre-registration Solves

Without pre-registration, an experimenter has enormous flexibility. They can:

• Try multiple primary metrics and report the one that is significant
• Try multiple segmentation cuts and report the one with the largest effect
• Try multiple statistical tests and report the one with the smallest p-value
• Stop the experiment early when the result looks good
• Change the hypothesis after seeing the data

Each of these choices inflates the false positive rate. If you try 20 metrics and report the one with p < 0.05, your actual false positive rate is not 5% — it is 64%.

Pre-registration does not eliminate this flexibility. It makes it visible. When you pre-register, you commit to a specific analysis plan. Deviations from the plan must be documented and justified.

What to Pre-register

A pre-registration document should contain:

1. Hypothesis: A specific, falsifiable prediction. "The new checkout flow will increase conversion by at least 2%" is a hypothesis. "The new checkout flow will improve the user experience" is not.

2. Primary metric: The single metric that will determine the decision. This must be specified before the experiment starts.

3. Secondary metrics: Additional metrics that provide context but do not drive the ship/no-ship decision.

4. Guardrail metrics: Metrics that must not degrade. These are safety checks.

5. Sample size and power analysis: How many users you need, what minimum detectable effect you are powering for, and what false positive rate you are willing to accept.

6. Analysis plan: The statistical test you will use, how you will handle outliers, and what segmentation you will report.

7. Stopping rule: When you will stop the experiment. Fixed sample size? Sequential testing with alpha spending?

Writing a Good Hypothesis

A good hypothesis has three properties:

• Specific: It names the metric and the direction of the effect.
• Falsifiable: It can be proven wrong by data.
• Decision-relevant: The result determines a real engineering or product decision.

Bad: "We believe the new recommendation algorithm will improve user engagement."

Good: "We believe the new recommendation algorithm will increase click-through rate on the homepage by at least 1.5% relative to the current algorithm, without decreasing session duration by more than 2%."

The good hypothesis specifies the metric (CTR), the direction (increase), the magnitude (1.5%), and a guardrail (session duration). This is what you need to make a ship/no-ship decision.

Pre-registration in Practice

Pre-registration does not require a formal registry. A document in your team's wiki or a comment in your experiment tracking tool is sufficient. The key is that it is written before the experiment starts and is not modified afterward.

# Experiment: New Checkout Flow

## Hypothesis
The new one-page checkout will increase purchase conversion by ≥2% relative to the current multi-page flow.

## Primary Metric
Purchase conversion rate (purchases / sessions)

## Secondary Metrics
- Average order value
- Time to complete checkout
- Cart abandonment rate

## Guardrail Metrics
- Page load time (must not increase by >200ms)
- Customer support tickets (must not increase by >5%)

## Sample Size
50,000 users per group (powered for 2% MDE at 80% power, α=0.05)

## Analysis Plan
Two-proportion z-test on conversion rate. Report 95% CI. Segment by new vs. returning users.

## Stopping Rule
Fixed sample size. No peeking.

When Pre-registration Is Not Enough

Pre-registration prevents intentional and unintentional p-hacking, but it does not prevent all forms of bias. If you pre-register 20 experiments and only publish the one that is significant, you have a publication bias problem. Pre-registration is necessary but not sufficient for honest inference.