Central Limit Theorem

The Statement

The Central Limit Theorem (CLT) states that the distribution of the sample mean approaches a normal distribution as the sample size increases, regardless of the shape of the population distribution. Formally:

If X₁, X₂, ..., Xₙ are independent and identically distributed random variables with mean μ and finite variance σ², then:

√n * (x̄ - μ) / σ  →  N(0, 1)   as n → ∞

Equivalently, the sample mean x̄ is approximately N(μ, σ²/n) for large n.

Why This Is Remarkable

The population distribution can be anything. It can be bimodal, heavily skewed, discrete, or bounded. The CLT says that if you take repeated samples and compute the mean of each, the histogram of those means will look like a bell curve. The shape of the original distribution washes out.

This is not intuitive. If session durations are exponentially distributed (most sessions are short, a few are very long), the distribution of the mean of 100 sessions is nearly symmetric and bell-shaped. The skewness does not disappear from the data, but it disappears from the sampling distribution of the mean.

What the CLT Enables

Almost all of parametric inference rests on the CLT:

• Confidence intervals for means use the normal (or t) distribution
• Hypothesis tests about means use z-statistics and t-statistics
• A/B test analysis assumes the difference in means is normally distributed
• Regression coefficients are asymptotically normal

Without the CLT, none of these tools would be valid for non-normal data. With the CLT, you can analyze revenue per user (heavily right-skewed), response times (log-normal), and conversion rates (binary) using the same normal-based machinery.

The Conditions

The CLT requires three conditions:

1. Independence: Sample observations must be independent. Violations occur with time series data, clustered samples, and repeated measurements on the same unit.

2. Identical distribution: Observations come from the same population. Mixing populations (e.g., combining data from two different user segments without accounting for the difference) violates this.

3. Finite variance: The population variance must be finite. Most real-world metrics satisfy this, but some heavy-tailed distributions (e.g., certain financial returns) may not.

When independence fails, the effective sample size is smaller than the nominal sample size, and the CLT approximation is optimistic. This is why clustered samples need special methods.

What the Sampling Distribution Is

The sampling distribution of the sample mean is the distribution of x̄ values you would obtain if you repeatedly drew samples of size n from the population and computed the mean each time. It is a theoretical construct, but you can approximate it by simulation.

The CLT tells us the shape of this distribution: approximately normal with mean μ and standard deviation σ/√n. That standard deviation has a name: the standard error of the mean (SEM).

SEM = σ / √n

The SEM quantifies how much the sample mean fluctuates from sample to sample. It decreases with the square root of n, which is why quadrupling the sample size only halves the standard error.

Simulation: Seeing the CLT in Action

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)

# Highly skewed population: exponential with mean 30
population = np.random.exponential(scale=30, size=100000)

# Draw 10,000 samples of size n and compute means
for n in [1, 5, 30, 100]:
    means = [np.mean(np.random.choice(population, size=n, replace=False))
              for _ in range(10000)]
    plt.hist(means, bins=50, alpha=0.5, label=f'n={n}')

plt.legend()
plt.title('Sampling Distribution of the Mean for Different n')
plt.show()

With n=1, the sampling distribution is identical to the population: exponential, heavily right-skewed. With n=5, the skew is reduced. With n=30, the distribution is visibly bell-shaped. With n=100, it is nearly indistinguishable from a normal distribution.

The Rate of Convergence

How large does n need to be? It depends on the population distribution:

• Symmetric, light-tailed distributions (uniform, normal): n = 10 to 15 is often sufficient
• Moderately skewed distributions (exponential, gamma): n = 30 is the classic rule of thumb
• Heavily skewed or multimodal distributions: n = 100 or more may be needed
• Binary outcomes (proportions): the rule is np ≥ 10 and n(1-p) ≥ 10

The rule of thumb n ≥ 30 comes from the exponential distribution case. It is not a universal threshold. For revenue data with extreme outliers, n = 30 may be far too small.

Standard Error in Practice

You almost never know σ, the population standard deviation. You estimate it with the sample standard deviation s:

Estimated SEM = s / √n

This estimation introduces additional uncertainty, which is why we use the t-distribution instead of the normal distribution for small samples. The t-distribution has heavier tails to account for the fact that s is itself a random variable.

Small Samples from Skewed Populations

The CLT is an asymptotic result. It describes what happens as n → ∞. For finite n, the approximation can be poor, especially in the tails of the distribution.

Example: You are analyzing customer lifetime value (CLV), which follows a Pareto-like distribution where the top 1% of customers generate 50% of revenue. With n = 20, the sampling distribution of the mean is still noticeably right-skewed. A normal-based confidence interval will undercover the true mean, particularly in the upper tail.

Heavy-Tailed Distributions

For distributions with infinite or very large variance (e.g., Cauchy distribution, some financial returns), the CLT does not apply in its standard form. The sample mean does not converge to a normal distribution, and the standard error is not a meaningful measure of spread.

In practice, most product metrics have finite variance, but the variance can be so large that the CLT approximation is poor at feasible sample sizes. Revenue per user during a viral event is a classic example.

Dependent Observations

The CLT requires independence. When observations are correlated, the effective sample size is smaller than n, and the true standard error is larger than σ/√n.

Common sources of dependence in analytical data:

• Time series: consecutive observations are correlated (autocorrelation)
• Clustered samples: users from the same household, city, or cohort are correlated
• Repeated measures: the same user observed on multiple days

The design effect quantifies the variance inflation:

True SE = (σ / √n) * √(design effect)

Ignoring dependence leads to confidence intervals that are too narrow and p-values that are too small. This is one of the most common errors in applied analytics.

Practical Diagnostics

Before applying normal-based inference:

1. Plot the data: A histogram or Q-Q plot reveals skewness and heavy tails
2. Check the sample size relative to the skewness: If the sample skewness is large, increase n or use a different method
3. Test for independence: For time series, compute the autocorrelation function. For clustered data, compute the intra-cluster correlation
4. Use simulation: Bootstrap the sampling distribution empirically and compare it to the normal approximation

from scipy import stats
import numpy as np

# Bootstrap the sampling distribution of the mean
np.random.seed(42)
data = np.random.exponential(scale=30, size=50)

bootstrap_means = [np.mean(np.random.choice(data, size=len(data), replace=True))
                   for _ in range(10000)]

# Compare to normal approximation
normal_approx = stats.norm(loc=data.mean(), scale=data.std() / np.sqrt(len(data)))

print(f"Bootstrap 95% CI: [{np.percentile(bootstrap_means, 2.5):.1f}, {np.percentile(bootstrap_means, 97.5):.1f}]")
print(f"Normal 95% CI:    [{normal_approx.ppf(0.025):.1f}, {normal_approx.ppf(0.975):.1f}]")

When the bootstrap and normal intervals diverge substantially, the CLT approximation is not reliable at this sample size.

Proportions as Means

A binary outcome (converted vs not converted) is a Bernoulli random variable with mean p and variance p(1-p). The sample proportion p̂ is the sample mean of these binary variables, so the CLT applies directly:

p̂ ~ N(p, p(1-p)/n)   approximately, for large n

The standard error of a proportion is:

SE(p̂) = √(p̂(1-p̂) / n)

The CLT approximation for proportions is reliable when np ≥ 10 and n(1-p) ≥ 10. For rare events (p < 0.01), this requires very large samples. For p = 0.001, you need n ≥ 10,000 for the normal approximation to be trustworthy.

Difference of Two Means

In A/B testing, you compare the mean of group A to the mean of group B. The difference x̄_A - x̄_B is also approximately normal, with:

Mean: μ_A - μ_B
SE: √(σ_A²/n_A + σ_B²/n_B)

This follows from the CLT applied to each group separately, plus the fact that the difference of two independent normal variables is normal.

Difference of Two Proportions

For conversion rate comparisons:

SE(p̂_A - p̂_B) = √(p̂_A(1-p̂_A)/n_A + p̂_B(1-p̂_B)/n_B)

This formula assumes the two samples are independent. In paired designs (same user exposed to both variants), the covariance term must be included.

Python: A/B Test Standard Error

import numpy as np

# A/B test results
n_A, conversions_A = 5000, 250
n_B, conversions_B = 5000, 300

p_A = conversions_A / n_A
p_B = conversions_B / n_B

# Standard error of the difference
se_diff = np.sqrt(p_A * (1 - p_A) / n_A + p_B * (1 - p_B) / n_B)

# Z-statistic for the difference
diff = p_B - p_A
z = diff / se_diff

print(f"Difference: {diff:.4f}")
print(f"SE: {se_diff:.4f}")
print(f"Z: {z:.2f}")

The CLT justifies treating this z-statistic as approximately standard normal, which allows you to compute p-values and confidence intervals for the difference.