Hypothesis Testing Framework

The Logic of Hypothesis Testing

Hypothesis testing is a formal procedure for using sample data to evaluate a claim about a population parameter. The framework inverts the question: instead of asking "what is the true effect?", we ask "if there were no effect, how surprising would our data be?"

This inversion is the entire engine of the method. We start by assuming the null hypothesis (H₀) — the position of no effect, no difference, no change. We then calculate a test statistic from our sample data. If the observed statistic is extreme enough under the assumption that H₀ is true, we reject H₀ in favor of the alternative hypothesis (H₁).

The Two Hypotheses

The null hypothesis (H₀) always represents the status quo or the "no effect" position. It is the hypothesis we attempt to reject. The alternative hypothesis (H₁ or Hₐ) represents the effect we are trying to detect.

H₀: μ = 100        (the mean conversion rate is 100)
H₁: μ ≠ 100       (the mean conversion rate differs from 100)

The alternative can be two-sided (≠), left-sided (<), or right-sided (>). The choice must be made before seeing the data. Choosing the direction after observing the data inflates the false positive rate because you are effectively cherry-picking the direction that favors significance.

Why Invert the Question?

You cannot directly prove an alternative hypothesis from sample data because sampling variability means almost any sample will show some deviation from the null. What you can do is quantify how unlikely your observation is if the null were true. If the probability is sufficiently small, the null becomes an implausible explanation for the data.

This is the same logic as a court trial: the defendant is presumed innocent (H₀), and the prosecution must present evidence so inconsistent with innocence that a reasonable jury rejects that presumption. The standard of proof — "beyond a reasonable doubt" — maps to the significance level in statistics.

The Four-Step Framework

Every hypothesis test follows the same structure:

1. State hypotheses. Define H₀ and H₁ in terms of population parameters.
2. Choose significance level (α). Set the threshold for how much evidence you need before rejecting H₀. Common choices are 0.05, 0.01, and 0.10.
3. Compute the test statistic and p-value. Calculate how far your sample result is from the null expectation, standardized by sampling variability.
4. Make a decision. Compare the p-value to α. If p ≤ α, reject H₀. Otherwise, fail to reject H₀.

The framework is universal. Whether you are testing a mean, a proportion, a variance, or a regression coefficient, the logic is identical. Only the test statistic and its sampling distribution change.

A Product Example

Suppose your team changes the checkout button color from gray to green. You want to know if the change increased the purchase rate.

H₀: p_new = p_old        (the change had no effect)
H₁: p_new > p_old       (the change increased purchase rate)

You collect data from 10,000 users in each group. The old flow converts at 3.2%, the new flow at 3.5%. Is this difference real or noise? The hypothesis test answers this by computing the probability of seeing a difference of 0.3 percentage points (or larger) if the true rates were identical.

Common Misconception: "Fail to Reject" ≠ "Accept"

Failing to reject H₀ does not mean H₀ is true. It means the data do not provide sufficient evidence against it. The sample size might be too small, the effect might be too small relative to noise, or the effect might genuinely be zero. The framework is asymmetric: rejecting H₀ is a strong claim; failing to reject is a weak claim.

This distinction matters in practice. When an A/B test shows no significant difference, the correct conclusion is "we did not detect an effect," not "there is no effect." The difference between these statements is the entire subject of statistical power, which we cover in a later topic.

Test Statistics and Sampling Distributions

A test statistic is a single number that summarizes the discrepancy between your sample data and the null hypothesis. It standardizes the observed effect by the amount of variability you would expect from random sampling alone.

The general form is:

test statistic = (observed value - null value) / standard error

The numerator measures how far your sample is from the null expectation. The denominator measures how much variation you would expect just from random chance. The ratio tells you whether the observed deviation is large relative to noise.

The Z-Statistic

When the sampling distribution of the sample mean is approximately normal (guaranteed by the Central Limit Theorem for sufficiently large samples), the test statistic follows a standard normal distribution under H₀:

Z = (x̄ - μ₀) / (σ / √n)

Where x̄ is the sample mean, μ₀ is the null hypothesis value, σ is the population standard deviation, and n is the sample size.

import numpy as np
from scipy import stats

sample_mean = 103.5
null_mean = 100
pop_std = 15
n = 200

z_stat = (sample_mean - null_mean) / (pop_std / np.sqrt(n))
p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))  # two-sided

print(f"Z = {z_stat:.3f}")
print(f"p-value = {p_value:.4f}")

The T-Statistic

When the population standard deviation is unknown (the common case), you estimate it from the sample. This adds uncertainty, and the test statistic follows a t-distribution with n-1 degrees of freedom instead of a normal distribution:

t = (x̄ - μ₀) / (s / √n)

Where s is the sample standard deviation. The t-distribution has heavier tails than the normal distribution, which means extreme values are more likely. As sample size grows, the t-distribution converges to the normal distribution.

from scipy import stats

sample = np.array([23, 28, 31, 26, 29, 35, 27, 30, 24, 32])
null_mean = 25

t_stat, p_value = stats.ttest_1samp(sample, null_mean)
print(f"t = {t_stat:.3f}, p = {p_value:.4f}")

Why Standardization Matters

Consider two experiments. In Experiment A, you observe a mean of 105 with a standard error of 2. In Experiment B, you observe a mean of 105 with a standard error of 10. The observed deviation from a null of 100 is the same (5 units), but the evidence against H₀ is very different.

Experiment A: Z = 5/2 = 2.5 → strong evidence against H₀
Experiment B: Z = 5/10 = 0.5 → weak evidence against H₀

Standardization accounts for sample size and variability. A large deviation means little if the data are noisy. A small deviation means a lot if the data are precise. The test statistic captures this tradeoff in a single number.

Choosing the Right Test Statistic

The choice depends on what parameter you are testing and what you know about the population:

The key insight is that the test statistic is not arbitrary. It is constructed so that its sampling distribution under H₀ is known. This is what allows you to compute p-values. Without a known reference distribution, you cannot calibrate how extreme your observation is.

Degrees of Freedom

Degrees of freedom (df) represent the number of independent pieces of information available to estimate variability. For a one-sample t-test, df = n-1 because one degree of freedom is consumed by estimating the sample mean.

The t-distribution's shape depends on df. With few degrees of freedom, the tails are heavy — extreme values are more probable. As df increases, the t-distribution approaches the standard normal. For df > 30, the difference is negligible for most practical purposes, but for small samples, using the normal approximation instead of the t-distribution will produce p-values that are too small and inflate your false positive rate.

P-Values and Significance Levels

The p-value is the probability of observing a test statistic as extreme as, or more extreme than, the one computed from your sample, assuming the null hypothesis is true. It is a measure of surprise: how surprising is your data if there really is no effect?

p-value = P(observed or more extreme data | H₀ is true)

A small p-value means the data are inconsistent with the null. A large p-value means the data are consistent with the null. The p-value is not the probability that H₀ is true, and it is not the probability that the result is due to chance. These are common misinterpretations.

Computing P-Values

The p-value comes from the sampling distribution of the test statistic under H₀. For a two-sided Z-test:

from scipy import stats
import numpy as np

z_stat = 2.15
p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))
print(f"p-value = {p_value:.4f}")
# Output: p-value = 0.0316

For a one-sided test, you use only one tail:

# Right-sided: H₁: μ > μ₀
p_value_right = 1 - stats.norm.cdf(z_stat)

# Left-sided: H₁: μ < μ₀
p_value_left = stats.norm.cdf(z_stat)

The Significance Level (α)

The significance level is the threshold you set before the experiment. It defines how much evidence you require before rejecting H₀. If p ≤ α, the result is "statistically significant."

α = 0.05  →  You accept a 5% chance of a false positive
α = 0.01  →  You accept a 1% chance of a false positive
α = 0.10  →  You accept a 10% chance of a false positive

The choice of α is a tradeoff between two types of error. A lower α reduces false positives (Type I errors) but increases false negatives (Type II errors). The right choice depends on the cost of each error type in your specific context.

Choosing α in Practice

In product experimentation, α = 0.05 is the default. It is a convention, not a law of nature. Consider adjusting it when:

• False positives are expensive. If a false positive leads to shipping a harmful change, use α = 0.01 or lower.
• False negatives are expensive. If missing a real effect means losing significant revenue, use α = 0.10.
• You are running many tests. If you test 20 variants, expect one false positive at α = 0.05 by chance alone. Lower α or apply multiple comparison corrections.

Statistical vs. Practical Significance

A result can be statistically significant but practically meaningless. With a sample of 1,000,000 users, a difference of 0.01 percentage points in conversion rate might yield p < 0.001. The effect is real but too small to matter.

Always report the effect size alongside the p-value:

Result: +0.3 percentage points (95% CI: [0.1, 0.5]), p = 0.003

The confidence interval tells you the range of plausible effect sizes. The p-value tells you whether the effect is distinguishable from zero. Both are needed for sound decision-making.

The P-Value Is Not the Probability H₀ Is True

This deserves emphasis because it is the most common misinterpretation in applied statistics. A p-value of 0.03 does not mean there is a 3% chance the null is true. It means that if the null were true, you would observe data this extreme 3% of the time.

The probability that H₀ is true given the data requires Bayesian reasoning with a prior probability. The p-value is a frequentist measure that conditions on H₀ being true. These are fundamentally different questions, and conflating them leads to overconfident conclusions.

Decision Rules

The decision rule is mechanical:

• If p ≤ α: reject H₀. The result is statistically significant.
• If p > α: fail to reject H₀. The result is not statistically significant.

Do not use phrases like "approaching significance" or "marginally significant" for p-values slightly above α. The threshold is a decision boundary, not a continuum. A p-value of 0.051 is not meaningfully different from 0.049 in terms of evidence, but the decision framework requires a binary choice. Report the exact p-value and let stakeholders weigh the practical implications.

Applying the Framework to A/B Tests

A/B testing is the most common application of hypothesis testing in product work. The framework translates directly: the control group represents the null hypothesis (current behavior), and the treatment group represents the alternative (proposed change).

Setting Up the Test

Before launching the experiment, you must define:

1. The metric. What are you measuring? Conversion rate, revenue per user, session duration?
2. The hypotheses. What effect are you testing for?
3. The significance level. What false positive rate can you tolerate?
4. The minimum detectable effect. What effect size would be practically meaningful?
5. The sample size. How many observations do you need?

Skipping any of these steps leads to uninterpretable results. A test without a pre-specified metric invites cherry-picking. A test without a power calculation is a coin flip.

A Complete Example

You are testing a new onboarding flow. The current flow has a 30% completion rate. You want to know if the new flow improves it.

import numpy as np
from scipy import stats

np.random.seed(42)

# Simulated data: 5000 users per group
control = np.random.binomial(1, 0.30, 5000)
treatment = np.random.binomial(1, 0.32, 5000)

# Two-proportion z-test
p_control = control.mean()
p_treatment = treatment.mean()
p_pooled = (control.sum() + treatment.sum()) / (len(control) + len(treatment))

se = np.sqrt(p_pooled * (1 - p_pooled) * (1/len(control) + 1/len(treatment)))
z_stat = (p_treatment - p_control) / se
p_value = 1 - stats.norm.cdf(z_stat)  # one-sided: H₁: p_treatment > p_control

print(f"Control: {p_control:.4f}")
print(f"Treatment: {p_treatment:.4f}")
print(f"Difference: {p_treatment - p_control:.4f}")
print(f"Z = {z_stat:.3f}, p = {p_value:.4f}")

Interpreting the Result

Suppose the output shows p = 0.012 with α = 0.05. You reject the null hypothesis. The new flow's higher completion rate is unlikely to be due to random chance alone.

But before shipping, ask:

• Is the effect size meaningful? A 0.1 percentage point increase might be statistically significant with enough users but not worth the engineering cost.
• Is the result robust? Check for novelty effects, segment interactions, and sensitivity to outliers.
• Was the analysis plan followed? If you peeked at the data multiple times and stopped when p < 0.05, the p-value is invalid.

Peeking and Optional Stopping

One of the most damaging practices in A/B testing is checking results daily and stopping the test as soon as p < 0.05. This inflates the false positive rate dramatically. If you check 20 times, the probability of at least one false positive is approximately 1 - (1 - 0.05)^20 ≈ 0.64, not 0.05.

Solutions:

• Fixed-horizon testing. Decide the sample size in advance and analyze only at the end.
• Sequential testing. Use methods like mSPRT (mixture Sequential Probability Ratio Test) that adjust for continuous monitoring.
• Always valid p-values. Use techniques like the e-values framework that remain valid under optional stopping.

Reporting Results

A complete A/B test report includes:

Metric: Onboarding completion rate
Control: 30.0% (n=5000)
Treatment: 32.0% (n=5000)
Absolute difference: +2.0 pp
Relative difference: +6.7%
95% CI: [+0.4, +3.6] pp
p-value: 0.012
Decision: Reject H₀ at α = 0.05

The confidence interval and p-value together tell the full story. The interval shows the range of plausible effect sizes. The p-value shows whether the effect is distinguishable from zero. Neither alone is sufficient for a sound decision.