Sequential Testing and Peeking

The Peeking Problem

Peeking is the practice of checking experiment results before the planned sample size is reached and stopping the experiment if the result is statistically significant. It is one of the most common and most damaging mistakes in A/B testing.

Why Peeking Inflates False Positives

A standard hypothesis test controls the false positive rate at a single point in time. When you run a test at α = 0.05 with a fixed sample size, you accept a 5% chance of a false positive. This guarantee holds only if you look at the data exactly once.

When you peek — checking the p-value every day, for example — you are running multiple tests. Each peek is an opportunity to stop and declare significance. The more you peek, the more opportunities you have for a false positive.

Consider an experiment with a true null hypothesis (no effect). You check the p-value daily for 14 days. On each day, there is a 5% chance of a false positive. The probability of at least one false positive across 14 independent tests is:

P(at least one false positive) = 1 - (1 - 0.05)^14 = 1 - 0.488 = 0.512

A 51% false positive rate. More than half the time, you would conclude there is an effect when there is none.

The Mathematics of Peeking

The p-value under the null hypothesis is uniformly distributed between 0 and 1. This means that at any point in time, P(p < 0.05) = 0.05. But the p-value is not independent across time — it is a random walk that drifts as data accumulates.

The probability that the p-value crosses 0.05 at some point during the experiment is much higher than 5%. This is because the p-value can cross 0.05, come back above, cross again, and so on. Each crossing is a false positive opportunity.

import numpy as np
from scipy import stats

np.random.seed(42)
num_simulations = 10000
num_peeks = 14
false_positive_rate = 0.0

for _ in range(num_simulations):
    # Simulate a true null: no difference between groups
    n_per_group = 5000
    control = np.random.binomial(1, 0.10, n_per_group)
    treatment = np.random.binomial(1, 0.10, n_per_group)
    
    # Peek daily (simulate by checking at fractions of the data)
    for day in range(1, num_peeks + 1):
        fraction = day / num_peeks
        n = int(n_per_group * fraction)
        c = control[:n]
        t = treatment[:n]
        _, p_value = stats.ttest_ind(t, c)
        if p_value < 0.05:
            false_positive_rate += 1
            break

print(f"False positive rate with {num_peeks} peeks: {false_positive_rate/num_simulations:.3f}")

This simulation shows that peeking 14 times inflates the false positive rate from 5% to approximately 30-50%, depending on the correlation structure.

Why People Peek

Peeking is natural. You are excited about the experiment. You want to know if it is working. You check the dashboard and see p = 0.03. You stop the experiment and declare victory.

The problem is that p = 0.03 on day 3 does not mean the same thing as p = 0.03 on day 14. Early p-values are noisy. A p-value of 0.03 on day 3 might be 0.30 on day 7 and 0.01 on day 14. The early significance is often a fluke.

The Only Safe Peek

There is exactly one safe way to peek: use a sequential testing procedure that adjusts the significance threshold to account for the number of looks. This is the subject of the next chapter.

Without a sequential procedure, the only safe approach is to commit to a fixed sample size and a single analysis at the end. No peeking. No early stopping. No exceptions.

Sequential Testing with Alpha Spending

Sequential testing allows you to peek at experiment results while controlling the false positive rate. The key idea is to spend your alpha budget carefully across multiple looks, so that the total false positive rate never exceeds the nominal level.

The Alpha Budget

Think of α = 0.05 as a budget. In a fixed-sample test, you spend the entire budget at one point in time. In a sequential test, you allocate portions of the budget to each interim analysis. The sum of all allocations must not exceed 0.05.

The allocation is defined by an alpha spending function α*(t), where t is the fraction of the total sample size observed so far. The spending function determines how much alpha you can use at each look.

Common Spending Functions

O'Brien-Fleming spending: Conservative early, liberal late. Most of the alpha is spent at the final analysis. This is appropriate when you want to avoid stopping early based on noisy data.

α*(t) = 2 - 2 × Φ(z_{α/2} / √t)

Pocock spending: Equal alpha at each look. More aggressive early stopping. This is appropriate when you want to stop as soon as possible.

α*(t) = α × ln(1 + (e - 1) × t)

Lan-DeMets spending: A flexible family that approximates O'Brien-Fleming or Pocock depending on parameters.

How Alpha Spending Works in Practice

Suppose you plan 4 interim analyses at 25%, 50%, 75%, and 100% of the sample size. You use O'Brien-Fleming spending with α = 0.05.

At the first look, you need a z-value of 4.05 (p < 0.0001) to stop. This is extremely conservative — you will almost never stop early. At the final look, the threshold is the standard 1.96 (p < 0.05).

from scipy.stats import norm
import math

def obrien_fleming_threshold(look_fraction: float, alpha: float = 0.05) -> float:
    z_alpha = norm.ppf(1 - alpha / 2)
    threshold = z_alpha / math.sqrt(look_fraction)
    return threshold

for t in [0.25, 0.50, 0.75, 1.00]:
    z_threshold = obrien_fleming_threshold(t)
    p_threshold = 2 * (1 - norm.cdf(z_threshold))
    print(f"Look at {t*100:.0f}%: z = {z_threshold:.2f}, p = {p_threshold:.4f}")

Group Sequential Design

A group sequential design pre-specifies:

1. Number of looks: How many times you will check the data.
2. Timing of looks: At what fraction of the sample size each look occurs.
3. Spending function: How alpha is allocated across looks.
4. Stopping rules: Efficacy boundary (stop for success) and futility boundary (stop for failure).

The design produces a set of critical values — one for each look. If the test statistic exceeds the critical value at any look, you stop and declare significance.

Futility Stopping

Sequential testing also allows for futility stopping — stopping the experiment early because the treatment is unlikely to work. This is not about controlling false positives; it is about saving resources.

If the observed effect at an interim look is so small that even with the full sample size you would not detect it, you can stop for futility. This does not inflate the false positive rate because you are not declaring a positive result.

Futility boundaries are typically based on conditional power — the probability of detecting an effect given the data observed so far. If conditional power falls below 20%, stopping for futility is reasonable.

Practical Implementation

Most experiment platforms support sequential testing. The implementation typically:

1. Pre-computes the critical values for each look based on the spending function.
2. At each look, compares the test statistic to the critical value.
3. Stops the experiment if the critical value is crossed.
4. Reports the adjusted p-value and confidence interval.

If you are implementing sequential testing yourself, use the gsDesign library in R or the sequential package in Python. Do not try to implement alpha spending from scratch — the mathematics is subtle and errors are easy to make.

Always Valid Inference and E-Vals

A newer approach to the peeking problem produces p-values that are valid at every point in time, without pre-specifying a spending function or a maximum sample size. These are called always valid p-values or e-values.

The Limitation of Group Sequential Designs

Group sequential designs require you to pre-specify the number of looks and their timing. If you deviate from the plan — add an extra look, change the timing — the error guarantees no longer hold.

In practice, experimenters often want flexibility. They want to check the data when a key stakeholder asks, not when the pre-specified schedule says. This flexibility breaks the group sequential guarantees.

E-Values: A Different Approach

An e-value is a non-negative random variable whose expected value under the null hypothesis is at most 1. You can think of it as a bet against the null: if the null is true, you expect to break even or lose money.

The key property of e-values is that you can multiply them across time and the product remains a valid e-value. This means you can peek as often as you want, combine the evidence, and always have a valid test.

E_combined = E_1 × E_2 × E_3 × ... × E_k

If the product exceeds 1/α, you reject the null at level α. This holds for any number of peeks, at any times, with any data-dependent stopping rule.

The E-Value for a Z-Test

For a z-test, the e-value at each look is:

E = exp(z × λ - λ²/2)

Where z is the test statistic and λ is a tuning parameter that controls the tradeoff between sensitivity to small and large effects.

import numpy as np
from scipy.stats import norm

def e_value(z: float, lambda_param: float = 1.0) -> float:
    return np.exp(z * lambda_param - lambda_param ** 2 / 2)

# Simulate peeking with e-values
np.random.seed(42)
num_simulations = 10000
num_peeks = 20
alpha = 0.05
false_positives = 0

for _ in range(num_simulations):
    n_per_group = 5000
    control = np.random.binomial(1, 0.10, n_per_group)
    treatment = np.random.binomial(1, 0.10, n_per_group)
    
    e_combined = 1.0
    for day in range(1, num_peeks + 1):
        fraction = day / num_peeks
        n = int(n_per_group * fraction)
        c = control[:n]
        t = treatment[:n]
        _, p_value = stats.ttest_ind(t, c)
        z = norm.ppf(1 - p_value / 2)
        e_combined *= e_value(z, lambda_param=0.5)
        
        if e_combined > 1 / alpha:
            false_positives += 1
            break

print(f"False positive rate with e-values ({num_peeks} peeks): {false_positives/num_simulations:.3f}")

Always Valid p-Values

An always valid p-value can be derived from an e-value:

p_always_valid = min(1, 1 / E)

This p-value is valid at every look, regardless of how many times you peek or when you peek. You can check it every hour, every day, or every week — the false positive rate is always controlled at α.

When to Use E-Values vs. Group Sequential

Use group sequential designs when you can commit to a pre-specified analysis schedule and want maximum power. Use e-values when you need flexibility and want guaranteed validity regardless of how you peek.

Designing Sequential Experiments

Choosing between a fixed-sample design and a sequential design is a business decision as much as a statistical one. This chapter provides a framework for making that choice.

When to Use a Fixed-Sample Design

Fixed-sample designs are appropriate when:

1. You can commit to not peeking. If your team has the discipline to wait for the full sample size, a fixed-sample design is simpler and more powerful.

2. The experiment is short. If the experiment runs for one week, the temptation to peek is lower, and the benefit of early stopping is smaller.

3. You need maximum power. Fixed-sample designs are more powerful than sequential designs at the same sample size because they do not spend alpha on interim looks.

4. The cost of a false positive is very high. If a false positive would lead to a costly engineering investment, the conservative nature of a fixed-sample design is valuable.

When to Use a Sequential Design

Sequential designs are appropriate when:

1. The experiment is long. If the experiment runs for months, the ability to stop early saves significant time.

2. The treatment has potential for serious harm. If the treatment could degrade the user experience, you want to stop as soon as you detect harm.

3. Traffic is limited. If you cannot reach the required sample size quickly, sequential designs allow you to stop early if the effect is large.

4. Stakeholders will peek regardless. If you cannot prevent peeking, use a sequential design that makes peeking safe.

Sample Size Inflation for Sequential Designs

Sequential designs require a larger maximum sample size than fixed-sample designs to achieve the same power. The inflation factor depends on the number of looks and the spending function:

The inflation is modest for a small number of looks. For 4 looks, you need only 4% more users than a fixed-sample design.

A Practical Sequential Testing Workflow

1. Define the maximum sample size using standard power analysis.

2. Inflate the sample size by the appropriate factor for your number of looks.

3. Choose a spending function. O'Brien-Fleming is the default choice — it is conservative early and powerful late.

4. Pre-compute the critical values for each look.

5. At each look, compare the test statistic to the critical value. Stop if the critical value is crossed.

6. Report the result using the sequential p-value or confidence interval.

Common Mistakes in Sequential Testing

Mistake 1: Peeking without a sequential procedure. This is the most common mistake. If you are going to peek, use a sequential procedure. If you are not going to peek, use a fixed-sample design. Do not peek in a fixed-sample design.

Mistake 2: Changing the spending function after seeing the data. The spending function must be chosen before the experiment starts. Changing it after seeing the data invalidates the error guarantees.

Mistake 3: Ignoring the futility boundary. If you only define an efficacy boundary, you may continue an experiment that is doomed to fail. Define both boundaries.

Mistake 4: Not accounting for the correlation between looks. The test statistics at different looks are correlated because they use overlapping data. Sequential procedures account for this correlation. Naive Bonferroni correction does not — it is too conservative.

The Bottom Line

Peeking without a sequential procedure is the single most damaging statistical error in A/B testing. It inflates the false positive rate from 5% to 30% or higher. The fix is simple: either commit to not peeking, or use a sequential testing procedure that makes peeking safe. There is no middle ground.