Type Errors and Statistical Power

Type I and Type II Errors

Every hypothesis test can produce two types of error. Understanding these errors is essential for designing experiments and interpreting results correctly.

The Two Errors

Type I error (false positive): Rejecting H₀ when it is actually true. You conclude there is an effect when there is none.

Type II error (false negative): Failing to reject H₀ when it is actually false. You miss a real effect.

                    | H₀ is True | H₀ is False
--------------------|------------|------------
Reject H₀           | Type I (α) | Correct (1-β)
Fail to reject H₀   | Correct    | Type II (β)

The Court Trial Analogy

• Type I error: Convicting an innocent person. The justice system treats this as especially grave, hence "beyond a reasonable doubt."
• Type II error: Acquitting a guilty person. Also bad, but the system accepts this risk to avoid Type I errors.

In product testing, the relative costs differ:

• Type I error cost: Shipping a change that does not work (or is harmful). Wasted engineering resources, potential user harm.
• Type II error cost: Missing a real improvement. Lost revenue, slower growth.

The right balance depends on your context. A medical trial prioritizes avoiding Type I errors (α = 0.01 or lower). A growth experiment might tolerate more Type I errors to avoid missing real effects (α = 0.10).

α and β Are Not Symmetric

α (the Type I error rate) is set directly by the researcher. Choose α = 0.05, and you have a 5% false positive rate.

β (the Type II error rate) is not set directly. It depends on:

• The true effect size
• The sample size
• The variance in the data
• The chosen α level

Power = 1 - β is the probability of correctly rejecting H₀ when it is false. A test with 80% power has a 20% chance of missing a real effect.

The Tradeoff

For a fixed sample size, decreasing α increases β. If you make it harder to reject H₀ (lower α), you also make it harder to detect real effects (higher β).

α = 0.01  →  stricter  →  more Type II errors  →  lower power
α = 0.10  →  lenient   →  fewer Type II errors  →  higher power

The only way to reduce both error rates simultaneously is to increase the sample size. More data provides more information, making it easier to distinguish signal from noise.

A Product Example

You test a new feature that truly increases conversion by 2 percentage points. You run the test with α = 0.05 and n = 500 per group.

• Type I error scenario: The feature does not work, but your test shows p = 0.03. You ship a useless change.
• Type II error scenario: The feature works (+2 pp), but your test shows p = 0.12. You discard a valuable improvement.

With n = 500 and a true effect of 2 pp, the power might be only 40%. You have a 60% chance of missing the effect. This is why sample size planning matters.

Why "Not Significant" Is Not "No Effect"

A common misinterpretation: p = 0.15 means there is no effect. Wrong. It means the data are insufficient to distinguish the effect from noise. The effect might be real but small, or the sample might be too small to detect it.

The correct interpretation: "We did not detect an effect at the α = 0.05 level." This leaves open the possibility that a larger sample or a more precise measurement would detect it.

Statistical Power

Statistical power is the probability that a test correctly rejects H₀ when a specific alternative hypothesis is true. It is the probability of detecting an effect when one exists.

Power = P(reject H₀ | H₁ is true) = 1 - β

Factors That Affect Power

1. Effect size. Larger effects are easier to detect. A 10 percentage point difference in conversion rate is obvious; a 0.1 percentage point difference requires enormous samples.

2. Sample size. More observations reduce sampling variability, making it easier to distinguish signal from noise. Power increases with √n.

3. Significance level (α). A higher α makes it easier to reject H₀, increasing power but also increasing the false positive rate.

4. Variability. Noisier data (higher variance) makes it harder to detect effects. Reducing measurement noise increases power without increasing sample size.

5. One-sided vs. two-sided tests. One-sided tests have more power for effects in the expected direction because all of α is in one tail.

Computing Power

from statsmodels.stats.power import TTestIndPower
import numpy as np

power_analysis = TTestIndPower()

# Parameters
effect_size = 0.5        # Cohen's d
alpha = 0.05
power = 0.80

# Calculate required sample size per group
n = power_analysis.solve_power(
    effect_size=effect_size,
    alpha=alpha,
    power=power,
    ratio=1.0,           # equal group sizes
    alternative='two-sided'
)

print(f"Required sample size per group: {n:.0f}")
print(f"Total sample size: {2*n:.0f}")

Power Analysis in Practice

Before running an experiment, conduct a power analysis to determine the required sample size. You need to specify:

1. Minimum detectable effect (MDE). The smallest effect size you care about. Smaller MDE → larger required sample.
2. Desired power. Typically 0.80 (80%). Higher power → larger required sample.
3. Significance level. Typically 0.05. Lower α → larger required sample.
4. Expected variance. From historical data or pilot studies.

from statsmodels.stats.power import TTestIndPower

power_analysis = TTestIndPower()

# Scenario: detect a 2 pp difference in conversion rate
# Baseline conversion: 10%, expected: 12%
# Cohen's h for proportions
from statsmodels.stats.proportion import proportion_effectsize

h = proportion_effectsize(0.10, 0.12)  # effect size for proportions
print(f"Cohen's h: {h:.4f}")

n = power_analysis.solve_power(
    effect_size=h,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative='two-sided'
)
print(f"Required sample size per group: {n:.0f}")

The Power Curve

A power curve shows how power changes with effect size or sample size:

import matplotlib.pyplot as plt
import numpy as np
from statsmodels.stats.power import TTestIndPower

power_analysis = TTestIndPower()
effect_sizes = np.linspace(0.1, 1.0, 50)

for n in [50, 100, 200, 500]:
    powers = [power_analysis.power(effect_size=es, nobs1=n, alpha=0.05) for es in effect_sizes]
    plt.plot(effect_sizes, powers, label=f'n={n}')

plt.xlabel('Effect size (Cohen\'s d)')
plt.ylabel('Power')
plt.legend()
plt.axhline(y=0.8, color='gray', linestyle='--', alpha=0.5)
plt.show()

The curve shows that power increases with effect size and sample size. For small effects, you need large samples to achieve adequate power.

Why 80% Power?

The convention of 80% power (β = 0.20) comes from a cost-benefit analysis. It means:

• 20% chance of missing a real effect
• 5% chance of a false positive (at α = 0.05)

The ratio of β to α is 4:1, reflecting the idea that a false positive is about 4 times as costly as a false negative in many contexts. This ratio is not universal. Adjust it based on your specific costs.

Post-Hoc Power: A Misleading Practice

Computing power after observing a non-significant result ("post-hoc power") is misleading. If p > 0.05, the post-hoc power is necessarily low. It adds no information beyond the p-value.

Instead, report the minimum detectable effect given your sample size and α. This tells readers what effect sizes your experiment was capable of detecting.

Sample Size Calculation

Sample size calculation is the most practical application of power analysis. It answers: "How many observations do I need to detect an effect of a given size with a given level of confidence?"

The Formula

For a two-sample t-test with equal group sizes:

n = 2 * (z_{α/2} + z_β)² * σ² / δ²

Where:

• n = sample size per group
• z_{α/2} = critical value for the significance level (1.96 for α = 0.05)
• z_β = critical value for the desired power (0.84 for 80% power)
• σ = population standard deviation
• δ = minimum detectable effect

import numpy as np
from scipy import stats

def sample_size_ttest(mde, sigma, alpha=0.05, power=0.80):
    """Sample size per group for a two-sample t-test."""
    z_alpha = stats.norm.ppf(1 - alpha/2)
    z_beta = stats.norm.ppf(power)
    n = 2 * (z_alpha + z_beta)**2 * sigma**2 / mde**2
    return np.ceil(n)

# Example: detect a 5-unit difference with σ = 15
n = sample_size_ttest(mde=5, sigma=15)
print(f"Required sample size per group: {n}")
print(f"Total sample size: {2*n}")

Sample Size for Proportions

For A/B tests with binary outcomes (conversion rates), use the proportion formula:

n = (z_{α/2} + z_β)² * [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²

from statsmodels.stats.proportion import proportion_effectsize
from statsmodels.stats.power import NormalIndPower

# Detect an increase from 10% to 12%
p1, p2 = 0.10, 0.12
h = proportion_effectsize(p1, p2)

power_analysis = NormalIndPower()
n = power_analysis.solve_power(
    effect_size=h,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative='two-sided'
)
print(f"Sample size per group: {n:.0f}")

Practical Considerations

1. Attrition and data quality. If you expect 10% of users to be excluded (bots, incomplete sessions), inflate the sample size accordingly.

2. Multiple metrics. If you test multiple metrics, the sample size must be large enough for the metric requiring the most power (usually the one with the smallest expected effect or highest variance).

3. Sequential testing. If you plan to peek at results, standard sample size calculations are invalid. Use sequential design methods that account for optional stopping.

4. Minimum sample size rules. Some methods require minimum samples for asymptotic approximations to hold. For t-tests, n > 30 per group is a common rule of thumb.

Sensitivity Analysis

Because you rarely know the true effect size or variance, conduct a sensitivity analysis:

import numpy as np
from statsmodels.stats.power import TTestIndPower

power_analysis = TTestIndPower()

# Vary the minimum detectable effect
for mde in [2, 3, 5, 10]:
    n = power_analysis.solve_power(
        effect_size=mde/15,  # assuming σ = 15
        alpha=0.05,
        power=0.80,
        ratio=1.0
    )
    print(f"MDE = {mde}: n per group = {n:.0f}")

This shows how sample size requirements change with the effect size you want to detect. Smaller effects require disproportionately larger samples.

When You Cannot Achieve Adequate Power

Sometimes the required sample size is impractical. Options:

1. Increase the MDE. Accept that you will only detect larger effects.
2. Reduce variance. Use more precise metrics, control for covariates, or use paired designs.
3. Relax α. Accept a higher false positive rate (with appropriate corrections).
4. Use a one-sided test. If the direction is clear, this provides more power.
5. Run a longer experiment. More time = more users = more power.
6. Use Bayesian methods. These can provide useful inferences with smaller samples by incorporating prior information.

Designing Experiments with Error Rates in Mind

Error rates are not abstract statistical concepts. They are design parameters that should be chosen deliberately based on the costs of being wrong.

Choosing α

The significance level should reflect the cost of a false positive relative to a false negative:

Choosing Power

Power should reflect the cost of missing a real effect:

• High power (0.90-0.95): When missing an effect is very costly (e.g., a major product bet).
• Standard power (0.80): Default for most experiments.
• Lower power (0.60-0.70): Acceptable for exploratory analyses where follow-up experiments will validate findings.

The Error Rate Tradeoff in Practice

Consider a pricing experiment. You want to test whether a new price increases revenue.

• Type I error cost: You raise prices based on a false signal. Revenue drops, customers churn.
• Type II error cost: You miss a real revenue increase. You leave money on the table.

If customer churn is more costly than missed revenue, use a lower α (0.01). If the opportunity cost of inaction is higher, use a higher α (0.10).

Power Analysis Workflow

1. Define the minimum detectable effect (MDE)
   → What effect size would be practically meaningful?

2. Estimate the variance
   → Use historical data or run a pilot

3. Choose α and power
   → Based on the costs of Type I and Type II errors

4. Calculate the required sample size
   → Use power analysis formulas or software

5. Check feasibility
   → Can you collect enough data in a reasonable time?

6. If not feasible, adjust:
   → Increase MDE, reduce power, or extend the timeline

Common Mistakes

1. No power analysis. Running experiments with arbitrary sample sizes. You might collect too little data (low power) or too much (wasted resources).

2. Post-hoc power. Computing power after a non-significant result. This is circular: if p > 0.05, power is necessarily low.

3. Ignoring multiple comparisons. Testing 20 metrics without adjusting α inflates the false positive rate. Power analysis must account for the corrected α.

4. Assuming 80% power is always right. The right power level depends on the context. A safety-critical test needs higher power; a quick exploratory test might need less.

5. Confusing statistical and practical significance. A test can have 99% power to detect an effect that is too small to matter. Power analysis should target the minimum practically significant effect.

Reporting Error Rates

A complete experiment report includes:

Design parameters:
- Significance level: α = 0.05 (two-sided)
- Power: 1 - β = 0.80
- Minimum detectable effect: 2 percentage points
- Sample size: 5,000 per group (calculated via power analysis)

Results:
- Observed effect: +1.8 pp (95% CI: [+0.3, +3.3])
- p-value: 0.018
- Decision: Reject H₀

Interpretation:
The experiment was designed to detect effects of 2 pp or larger with
80% confidence. The observed effect of 1.8 pp is statistically
significant but slightly below the pre-specified MDE. The confidence
interval suggests the true effect is likely between 0.3 and 3.3 pp.

This level of transparency allows stakeholders to evaluate whether the experiment was properly designed and whether the results are actionable.