Sample Size and Power

The Power Analysis Framework

Power analysis is the process of determining how many users you need in an experiment to detect an effect of a given size with a given level of confidence. It is the bridge between the effect you care about and the experiment you can run.

The Four Parameters

Every power analysis involves four parameters. If you know three, you can calculate the fourth:

The significance level α is the probability of concluding there is an effect when there is none. Power (1 - β) is the probability of detecting an effect when it truly exists. The minimum detectable effect (MDE) is the smallest effect that would be practically meaningful.

Why Power Matters

An underpowered experiment is a waste of time. If your experiment has 30% power, you have a 70% chance of concluding there is no effect even when the treatment works. You might abandon a good treatment because your experiment was too small to detect its benefit.

An overpowered experiment is also a waste. If you need 10,000 users to detect a 5% effect but you run 100,000, you are exposing 90,000 extra users to an unproven treatment for no additional information.

Power analysis finds the sweet spot: the minimum sample size that gives you a reasonable chance of detecting the effect you care about.

The Sample Size Formula

For a two-proportion z-test comparing conversion rates between two groups, the required sample size per group is:

n = (z_{α/2} + z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²

Where:

• p₁ is the baseline conversion rate
• p₂ is the conversion rate under treatment (p₁ + δ)
• z_{α/2} is the critical value for the significance level
• z_β is the critical value for the desired power

from scipy.stats import norm
import math

def sample_size_two_proportions(baseline_rate: float, mde_absolute: float,
                                  alpha: float = 0.05, power: float = 0.80) -> int:
    p1 = baseline_rate
    p2 = baseline_rate + mde_absolute
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)
    
    numerator = (z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))
    denominator = (p2 - p1) ** 2
    n = numerator / denominator
    return math.ceil(n)

n = sample_size_two_proportions(baseline_rate=0.10, mde_absolute=0.005)
print(f"Required sample size per group: {n:,}")

The MDE-Sample Size Tradeoff

The relationship between MDE and sample size is inverse and quadratic. To detect an effect half as large, you need four times the sample size. This is the most important practical fact about power analysis.

This quadratic relationship means that detecting small effects is expensive. Before launching an experiment, you must decide: is the effect I am powering for small enough to matter? If a 0.25% improvement is not worth the engineering effort, do not run an experiment powered to detect it.

Practical Power Analysis

In practice, you usually work backward:

1. Define the MDE: What is the smallest effect that would change your decision? This is a business question, not a statistical question.

2. Set α and power: Standard values are α = 0.05 and power = 0.80. Some teams use α = 0.10 for faster iteration, accepting a higher false positive rate.

3. Estimate the baseline: Use historical data to estimate the current conversion rate. The baseline affects sample size because variance is maximized at p = 0.5.

4. Calculate sample size: Use the formula or a power analysis library.

5. Check feasibility: Can you reach the required sample size within a reasonable time? If not, increase the MDE or accept lower power.

Power Calculations in Practice

Power analysis is not a theoretical exercise. It is a practical tool that determines whether your experiment can produce a meaningful answer. This chapter walks through real-world power calculations and the decisions they inform.

Continuous Metrics: t-Test Power

When your primary metric is continuous (revenue per user, session duration), you use a two-sample t-test instead of a z-test. The sample size formula is:

n = (z_{α/2} + z_β)² × 2σ² / δ²

Where σ is the standard deviation of the metric and δ is the minimum detectable effect.

from scipy.stats import norm
import math

def sample_size_t_test(std_dev: float, mde_absolute: float,
                       alpha: float = 0.05, power: float = 0.80) -> int:
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)
    n = (z_alpha + z_beta) ** 2 * 2 * std_dev ** 2 / mde_absolute ** 2
    return math.ceil(n)

n = sample_size_t_test(std_dev=15.0, mde_absolute=1.0)
print(f"Required sample size per group: {n:,}")

The critical input is the standard deviation. You must estimate this from historical data. Underestimating σ leads to an underpowered experiment.

Using statsmodels for Power Analysis

The statsmodels library provides built-in power analysis functions that handle the details:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# For proportions
baseline = 0.10
mde_relative = 0.05  # 5% relative lift
p1 = baseline
p2 = baseline * (1 + mde_relative)
effect_size = proportion_effectsize(p1, p2)

analysis = NormalIndPower()
n = analysis.solve_power(effect_size=effect_size, alpha=0.05, power=0.80, ratio=1.0)
print(f"Required sample size per group: {math.ceil(n):,}")

The Baseline Rate Matters

For proportion metrics, the baseline rate affects the required sample size. Variance of a proportion is p(1-p), which is maximized at p = 0.5. This means:

• A metric with a 50% baseline requires the smallest sample size.
• A metric with a 1% baseline requires a much larger sample size.
• A metric with a 10% baseline is in between.

This is why experiments on rare events (e.g., purchases from landing page visitors) require enormous sample sizes.

Adjusting for Multiple Comparisons

If you are testing multiple treatments or multiple metrics, you must adjust your significance level to control the family-wise error rate. The Bonferroni correction divides α by the number of comparisons:

num_comparisons = 3
alpha_adjusted = 0.05 / num_comparisons  # 0.0167

n = sample_size_two_proportions(baseline_rate=0.10, mde_absolute=0.005,
                                  alpha=alpha_adjusted, power=0.80)
print(f"Required sample size per group (Bonferroni): {n:,}")

The Bonferroni correction is conservative. Less conservative alternatives include the Holm-Bonferroni method and the Benjamini-Hochberg procedure, which control the false discovery rate instead of the family-wise error rate.

When You Cannot Reach the Required Sample Size

Sometimes the required sample size exceeds your available traffic. You have three options:

1. Increase the MDE: Accept that you can only detect larger effects. This is often the right choice — if a small effect is not worth detecting, do not try to detect it.

2. Extend the experiment duration: Run the experiment longer to accumulate more users. Be aware of seasonality and novelty effects.

3. Reduce variance: Use techniques like CUPED (Controlled-experiment Using Pre-Experiment Data) to reduce the variance of your metric, which reduces the required sample size.

Do not simply accept lower power without acknowledging the consequence: you are more likely to miss a real effect. Document this tradeoff explicitly.

Variance Reduction and CUPED

Variance reduction techniques allow you to detect smaller effects with the same sample size, or the same effect with a smaller sample size. The most widely used technique in industry is CUPED (Controlled-experiment Using Pre-Experiment Data).

Why Variance Reduction Works

The power of an experiment depends on the signal-to-noise ratio. The signal is the treatment effect. The noise is the variance of the metric. You can increase power by increasing the signal (not in your control) or decreasing the noise (in your control).

CUPED decreases noise by using pre-experiment data to explain some of the variance in the outcome metric. If a user's pre-experiment behavior predicts their post-experiment behavior, you can subtract out the predictable portion and analyze only the residual.

The CUPED Method

CUPED works as follows:

1. Collect pre-experiment data: For each user, measure the same metric (or a correlated metric) during a pre-experiment period.

2. Compute the adjusted metric: For each user, calculate:

Y_cuped = Y - θ × (X - E[X])

Where Y is the outcome metric, X is the pre-experiment covariate, E[X] is the mean of X, and θ is the regression coefficient of Y on X.

3. Analyze the adjusted metric: Run your standard statistical test on Y_cuped instead of Y.

The optimal θ is the coefficient from regressing Y on X:

θ = Cov(Y, X) / Var(X)

import numpy as np
from scipy import stats

def cuped_adjust(y: np.ndarray, x: np.ndarray) -> np.ndarray:
    theta = np.cov(x, y)[0, 1] / np.var(x)
    y_cuped = y - theta * (x - np.mean(x))
    return y_cuped

np.random.seed(42)
n = 10000
x = np.random.normal(100, 15, n)  # pre-experiment metric
y = 0.7 * x + np.random.normal(0, 10, n)  # outcome metric correlated with x

y_cuped = cuped_adjust(y, x)
print(f"Original variance: {np.var(y):.2f}")
print(f"CUPED variance: {np.var(y_cuped):.2f}")
print(f"Variance reduction: {(1 - np.var(y_cuped)/np.var(y))*100:.1f}%")

How Much Variance Reduction to Expect

The variance reduction from CUPED depends on the correlation between the pre-experiment covariate and the outcome metric:

The variance reduction equals ρ². A correlation of 0.7 reduces variance by 49%, which means you need roughly half the sample size to achieve the same power.

Choosing a Covariate

The covariate must satisfy two conditions:

1. It is measured before the experiment starts. Using post-experiment data would introduce bias.

2. It is correlated with the outcome metric. The stronger the correlation, the greater the variance reduction.

Good covariates:

• The same metric measured during a pre-experiment period (e.g., pre-experiment conversion rate)
• A metric that is mechanically related to the outcome (e.g., page views as a covariate for time on page)
• User characteristics that predict behavior (e.g., historical purchase frequency)

Bad covariates:

• Metrics affected by the treatment (this would remove the treatment effect)
• Metrics measured after the experiment starts (this introduces bias)
• Metrics with no correlation to the outcome (no variance reduction)

CUPED in Practice

CUPED is used at Microsoft, Netflix, Google, and other companies running large-scale experiments. The typical implementation:

1. Define a pre-experiment window (e.g., 14 days before the experiment starts).
2. Compute the covariate for each user during this window.
3. Apply the CUPED adjustment to the outcome metric.
4. Run the standard statistical test on the adjusted metric.

The key insight is that CUPED does not change the expected value of the treatment effect. It only reduces variance. This means the point estimate of the effect is unbiased, but the confidence interval is narrower.

Common Power Analysis Mistakes

Power analysis is straightforward in theory but frequently done wrong in practice. These are the mistakes that invalidate experiment results.

Mistake 1: Powering for the Wrong Effect Size

The most common mistake is powering for an effect size that is too small. Teams often power for a 1% relative improvement because they want to detect any improvement. But a 1% improvement may not be worth the engineering effort to implement the treatment.

The MDE should be the smallest effect that would change your decision. If a 1% improvement is not actionable, power for 5% instead. This reduces the required sample size by a factor of 25.

Mistake 2: Ignoring the Baseline Rate

For proportion metrics, the baseline rate dramatically affects the required sample size. Powering for a 0.5% absolute improvement on a 10% baseline requires far fewer users than the same absolute improvement on a 1% baseline.

Always use the actual baseline rate from your historical data, not a round number or a guess.

Mistake 3: Not Accounting for Multiple Comparisons

If you test 5 metrics and declare success if any one is significant, your false positive rate is not 5% — it is 23%. You must adjust α for the number of comparisons.

from scipy.stats import norm
import math

def sample_size_with_comparisons(baseline_rate: float, mde_absolute: float,
                                  num_comparisons: int, alpha: float = 0.05,
                                  power: float = 0.80) -> int:
    alpha_adjusted = alpha / num_comparisons
    p1 = baseline_rate
    p2 = baseline_rate + mde_absolute
    z_alpha = norm.ppf(1 - alpha_adjusted / 2)
    z_beta = norm.ppf(power)
    numerator = (z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))
    denominator = (p2 - p1) ** 2
    return math.ceil(numerator / denominator)

n_uncorrected = sample_size_with_comparisons(0.10, 0.005, num_comparisons=1)
nbonferroni = sample_size_with_comparisons(0.10, 0.005, num_comparisons=5)
print(f"Uncorrected: {n_uncorrected:,} per group")
print(f"Bonferroni (5 comparisons): {n_bonferroni:,} per group")

Mistake 4: Peeking Without Adjustment

If you check your experiment results daily and stop when p < 0.05, your false positive rate is not 5% — it can be 30% or higher. This is the peeking problem, and it requires sequential testing methods to fix.

The fix is either:

• Commit to a fixed sample size and do not peek
• Use a sequential testing procedure with alpha spending

Mistake 5: Using the Wrong Variance Estimate

For continuous metrics, the standard deviation must be estimated from historical data. Using a guess or a value from a different context leads to an incorrect sample size.

Estimate the variance from at least 30 days of historical data. If the metric has strong day-of-week effects, estimate variance separately for each day of week.

Mistake 6: Not Accounting for Attrition

If users can drop out of the experiment (e.g., they uninstall the app), you need to inflate your sample size to account for attrition. If you expect 10% attrition, divide your calculated sample size by 0.9.

A Power Analysis Checklist

Before launching any experiment:

• MDE is defined and justified by business impact
• Baseline rate is estimated from historical data
• α and power are set explicitly
• Multiple comparisons are accounted for
• Variance reduction (CUPED) is considered
• Attrition is accounted for
• Feasibility is checked (can you reach the sample size in time?)
• The stopping rule is defined (fixed sample or sequential)