Building Confidence Intervals

The Definition

A confidence interval (CI) is a range of values, computed from sample data, that is likely to contain the true population parameter. The confidence level (typically 95%) quantifies how often the procedure produces intervals that capture the true parameter.

The correct interpretation: if you repeated the sampling process many times and computed a 95% CI each time, approximately 95% of those intervals would contain the true parameter.

The incorrect interpretation: there is a 95% probability that the true parameter lies within this specific interval. Once the interval is computed, the parameter is either in it or not. The probability statement is about the procedure, not the interval.

This distinction matters in practice. When you report "the 95% CI for mean session duration is [12.3, 15.7] minutes," you are describing the reliability of your estimation procedure, not the probability that the true mean is in that range.

The General Form

Every confidence interval has the same structure:

CI = point estimate ± margin of error
margin of error = critical value × standard error

The point estimate is your best guess (sample mean, sample proportion, difference in means). The standard error quantifies the variability of that estimate. The critical value comes from a reference distribution (normal or t) and determines how many standard errors you need for the desired confidence level.

For a 95% CI using the normal distribution, the critical value is 1.96. This means the interval extends 1.96 standard errors in each direction from the point estimate.

Why 95%?

The 95% convention is arbitrary but deeply embedded in practice. It corresponds to a 5% significance level, which has historical roots in R.A. Fisher's statistical tables. Other common levels are 90% (critical value 1.645) and 99% (critical value 2.576).

The choice of confidence level is a tradeoff between precision and coverage:

• 90% CI: narrower interval, but 10% of the time it misses the true parameter
• 95% CI: standard balance
• 99% CI: wider interval, but only 1% miss rate

In product analytics, 95% is the default. In safety-critical or high-stakes decisions, 99% may be appropriate. In exploratory analysis where you want to detect any signal, 90% can be useful.

Known Variance: The z-Interval

When the population standard deviation σ is known, the confidence interval for the mean is:

CI = x̄ ± z* × (σ / √n)

where z* is the critical value from the standard normal distribution. For 95% confidence, z* = 1.96.

In practice, σ is almost never known. You estimate it with the sample standard deviation s. When you substitute s for σ, the sampling distribution is no longer exactly normal. It follows a t-distribution with n-1 degrees of freedom.

Unknown Variance: The t-Interval

CI = x̄ ± t* × (s / √n)

where t* comes from the t-distribution with n-1 degrees of freedom. The t-distribution has heavier tails than the normal distribution, which accounts for the additional uncertainty from estimating σ.

As n increases, the t-distribution converges to the normal distribution. For n > 30, the difference is small. For n < 15, it is substantial.

Python Implementation

import numpy as np
from scipy import stats

np.random.seed(42)
data = np.random.normal(loc=50, scale=10, size=25)

n = len(data)
x_bar = data.mean()
s = data.std(ddof=1)
se = s / np.sqrt(n)

# 95% CI using t-distribution
t_crit = stats.t.ppf(0.975, df=n-1)
ci_lower = x_bar - t_crit * se
ci_upper = x_bar + t_crit * se

print(f"Sample mean: {x_bar:.2f}")
print(f"95% CI: [{ci_lower:.2f}, {ci_upper:.2f}]")

# Using scipy's built-in function
ci = stats.t.interval(0.95, df=n-1, loc=x_bar, scale=se)
print(f"Scipy CI: [{ci[0]:.2f}, {ci[1]:.2f}]")

Choosing Between z and t

The conservative choice is always the t-distribution. It gives slightly wider intervals when n is small and converges to the normal interval when n is large. There is no downside to using t when σ is unknown.

Sample Size Determination

Often you work backwards: you want a confidence interval of a certain width and need to know how large n must be.

n = (z* × σ / E)²

where E is the desired margin of error (half-width of the interval). For example, if you want a 95% CI for mean revenue per user with a margin of error of $0.50, and σ = $15:

n = (1.96 × 15 / 0.50)² = 3,457

You need at least 3,457 observations. This calculation is essential for experiment design and survey planning.

Single Proportion

For a binary outcome, the sample proportion p̂ is approximately normal with mean p and standard error √(p̂(1-p̂)/n). The confidence interval is:

CI = p̂ ± z* × √(p̂(1-p̂) / n)

This is the Wald interval. It works well when np̂ ≥ 10 and n(1-p̂) ≥ 10. For small samples or extreme proportions, the Wald interval can produce impossible values (below 0 or above 1) and has poor coverage.

The Wilson score interval is more robust:

center = (p̂ + z²/2n) / (1 + z²/n)
half_width = z * √(p̂(1-p̂)/n + z²/4n²) / (1 + z²/n)
CI = [center - half_width, center + half_width]

from statsmodels.stats.proportion import proportion_confint

# Wilson interval for 25 conversions out of 500
ci_low, ci_high = proportion_confint(count=25, nobs=500, alpha=0.05, method='wilson')
print(f"Wilson 95% CI: [{ci_low:.4f}, {ci_high:.4f}]")

Difference of Two Proportions

In A/B testing, you compare conversion rates between two groups. The confidence interval for the difference is:

CI = (p̂_A - p̂_B) ± z* × √(p̂_A(1-p̂_A)/n_A + p̂_B(1-p̂_B)/n_B)

If the interval contains 0, you cannot rule out that there is no difference between the groups. If the entire interval is above 0, group B has a statistically significant higher conversion rate.

Difference of Two Means

For continuous metrics (revenue, session duration), the confidence interval for the difference in means is:

CI = (x̄_A - x̄_B) ± t* × √(s_A²/n_A + s_B²/n_B)

The degrees of freedom for the t-distribution can be approximated using the Welch-Satterthwaite equation, which accounts for unequal variances and sample sizes:

df ≈ (s_A²/n_A + s_B²/n_B)² / [(s_A²/n_A)²/(n_A-1) + (s_B²/n_B)²/(n_B-1)]

from scipy import stats
import numpy as np

# A/B test: mean session duration
np.random.seed(42)
control = np.random.normal(120, 30, size=1000)
treatment = np.random.normal(125, 32, size=1000)

diff = treatment.mean() - control.mean()
se_diff = np.sqrt(control.var(ddof=1)/len(control) + treatment.var(ddof=1)/len(treatment))

# Welch's t-interval
df = se_diff**4 / ((control.var(ddof=1)/len(control))**2/(len(control)-1) + 
                   (treatment.var(ddof=1)/len(treatment))**2/(len(treatment)-1))
t_crit = stats.t.ppf(0.975, df)

ci = (diff - t_crit * se_diff, diff + t_crit * se_diff)
print(f"Difference: {diff:.2f} seconds")
print(f"95% CI: [{ci[0]:.2f}, {ci[1]:.2f}]")

Interpreting the Interval in Business Terms

A confidence interval is more informative than a p-value because it shows both the magnitude of the effect and the uncertainty around it.

• "The new feature increased session duration by 5.2 seconds (95% CI: [1.3, 9.1])" tells you the effect is likely positive and could be as large as 9 seconds or as small as 1 second.
• "The new feature increased session duration by 5.2 seconds (95% CI: [-0.8, 11.2])" tells you the effect might be zero, and you need more data to be sure.

The interval width is a direct measure of precision. A wide interval means you need more data. A narrow interval means you can make a confident decision.

Misinterpretation 1: The Parameter Is Random

Wrong: "There is a 95% probability that the true mean is between 12.3 and 15.7."

Right: "The procedure that produced this interval captures the true mean 95% of the time. This specific interval either contains the true mean or it does not."

The parameter is a fixed (unknown) value. The interval is random because it depends on the sample. The probability statement is about the procedure across repeated samples, not about this one interval.

Misinterpretation 2: The Interval Contains 95% of the Data

Wrong: "95% of users have session durations between 12.3 and 15.7 minutes."

Right: The confidence interval is about the population mean, not about individual observations. The interval for the mean is much narrower than the range of individual values because the standard error of the mean is σ/√n, not σ.

To describe the spread of individual values, use a prediction interval or simply report percentiles of the data.

Misinterpretation 3: Statistical Significance Equals Practical Importance

A 95% CI of [0.01, 0.03] seconds for a difference in page load time is statistically significant (does not contain 0) but practically irrelevant. A CI of [-2.5, 15.0] seconds is not statistically significant but includes effects that would be very important if real.

Always report the interval, not just whether it contains 0. The interval shows the range of plausible effect sizes, which is what decision-makers need.

Misinterpretation 4: Overlapping Intervals Mean No Significant Difference

Two groups can have overlapping 95% CIs while the difference between them is statistically significant. The correct approach is to compute the CI for the difference, not to compare the individual intervals.

# Wrong: comparing individual CIs
# CI_A = [10, 20], CI_B = [15, 25] -> overlap -> "not significant" (WRONG)

# Right: compute CI for the difference
# diff = 5, CI_diff = [1, 9] -> does not contain 0 -> significant

Multiple Comparisons

If you compute 20 confidence intervals at 95% confidence, you expect one of them to miss the true parameter by chance. When testing multiple variants or multiple metrics, the probability that at least one interval is wrong increases.

Corrections include the Bonferroni correction (divide α by the number of comparisons) or the Benjamini-Hochberg procedure (control the false discovery rate). In A/B testing, always pre-specify your primary metric and treat secondary metrics as exploratory.

Reporting Best Practices

1. Always report the point estimate, the interval, and the confidence level
2. State the sample size
3. For A/B tests, report the absolute difference and the relative difference with intervals
4. Include a plain-language interpretation: "We are 95% confident the true effect lies between X and Y"
5. When the interval is wide, recommend more data rather than making a decision on a fragile estimate