T-Tests in Practice

One-Sample T-Test

The one-sample t-test evaluates whether the mean of a single sample differs from a known or hypothesized population mean. It is the simplest t-test and the foundation for understanding the more complex variants.

When to Use It

Use a one-sample t-test when you have one group of observations and a specific reference value to compare against. Common scenarios:

• Is the average session duration different from 10 minutes?
• Is the mean customer satisfaction score above 4.0 (on a 5-point scale)?
• Does the average order value differ from $50?

The key requirement is a meaningful reference value. Without one, the test has no null hypothesis to evaluate.

The Test Statistic

t = (x̄ - μ₀) / (s / √n)

Where x̄ is the sample mean, μ₀ is the hypothesized population mean, s is the sample standard deviation, and n is the sample size. The statistic follows a t-distribution with n-1 degrees of freedom under H₀.

from scipy import stats
import numpy as np

# Session durations in minutes for 50 users
session_durations = np.array([
    8.2, 12.1, 9.5, 15.3, 7.8, 11.2, 10.5, 13.7, 9.1, 14.2,
    10.8, 8.9, 12.5, 11.0, 9.8, 13.1, 10.2, 8.5, 14.8, 11.5,
    9.3, 12.8, 10.0, 8.7, 13.5, 11.8, 9.6, 12.3, 10.4, 8.1,
    14.0, 11.3, 9.9, 12.6, 10.7, 8.4, 13.3, 11.1, 9.4, 12.9,
    10.1, 8.6, 13.8, 11.6, 9.7, 12.4, 10.6, 8.3, 13.0, 11.4
])

# H₀: μ = 10, H₁: μ ≠ 10
t_stat, p_value = stats.ttest_1samp(session_durations, popmean=10.0)

print(f"Sample mean: {session_durations.mean():.2f}")
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")
print(f"Degrees of freedom: {len(session_durations) - 1}")

Assumptions

The one-sample t-test relies on three assumptions:

1. Independence. Observations are independent of each other. Violated if you have repeated measures from the same user without accounting for the pairing.
2. Normality. The data are approximately normally distributed. The t-test is robust to mild violations with n > 30 due to the Central Limit Theorem, but severe skewness or outliers can distort results with small samples.
3. Random sampling. The sample represents the population you want to infer about.

Check normality with a Shapiro-Wilk test or a Q-Q plot:

from scipy import stats

stat, p = stats.shapiro(session_durations)
print(f"Shapiro-Wilk p-value: {p:.4f}")
# p > 0.05 suggests normality is plausible

Interpreting the Output

The output gives you a t-statistic and a p-value. The t-statistic tells you how many standard errors the sample mean is from the hypothesized mean. The p-value tells you how likely that distance is under the null.

A t-statistic of 2.5 with 49 degrees of freedom means the sample mean is 2.5 standard errors above the null value. The corresponding p-value (two-sided) is approximately 0.016. At α = 0.05, you reject the null.

Confidence Interval Approach

The t-test is equivalent to checking whether the confidence interval contains the null value. A 95% CI that excludes 10 corresponds to p < 0.05 in a two-sided test.

from scipy import stats
import numpy as np

n = len(session_durations)
mean = session_durations.mean()
sem = stats.sem(session_durations)  # standard error of the mean
ci = stats.t.interval(0.95, df=n-1, loc=mean, scale=sem)

print(f"95% CI: [{ci[0]:.2f}, {ci[1]:.2f}]")

If the interval does not contain 10, the test rejects H₀. The interval is more informative than the p-value because it shows the range of plausible values for the true mean.

Two-Sample T-Test (Independent Samples)

The two-sample t-test compares the means of two independent groups. This is the workhorse of A/B testing: control versus treatment, old versus new, variant A versus variant B.

When to Use It

Use a two-sample t-test when:

• You have two separate groups of observations
• You want to know if their population means differ
• The groups are independent (no natural pairing between observations)

Examples:

• Compare average revenue per user between two pricing pages
• Compare click-through rates between two email subject lines
• Compare task completion times between two interface designs

The Test Statistic

When you assume equal variances in both groups (Student's t-test):

t = (x̄₁ - x̄₂) / (s_p * √(1/n₁ + 1/n₂))

Where s_p is the pooled standard deviation:

s_p = √[((n₁-1)s₁² + (n₂-1)s₂²) / (n₁ + n₂ - 2)]

When you do not assume equal variances (Welch's t-test):

t = (x̄₁ - x̄₂) / √(s₁²/n₁ + s₂²/n₂)

Welch's t-test uses a more complex degrees of freedom calculation (the Welch-Satterthwaite equation) and is the safer default because it performs well whether or not variances are equal.

from scipy import stats
import numpy as np

np.random.seed(42)
control = np.random.normal(100, 15, 200)    # mean=100, sd=15, n=200
treatment = np.random.normal(105, 18, 200)  # mean=105, sd=18, n=200

# Welch's t-test (does not assume equal variances)
t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)

print(f"Control mean: {control.mean():.2f}")
print(f"Treatment mean: {treatment.mean():.2f}")
print(f"Difference: {treatment.mean() - control.mean():.2f}")
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

Equal Variance Assumption

Student's t-test assumes both groups have the same population variance. When this holds, it is slightly more powerful than Welch's t-test. When it does not hold, Student's t-test can produce misleading p-values.

Check the assumption with Levene's test:

from scipy import stats

stat, p = stats.levene(control, treatment)
print(f"Levene's test p-value: {p:.4f}")
# p > 0.05 suggests equal variances is plausible

In practice, use Welch's t-test by default. The power difference is negligible when variances are truly equal, and Welch's protects you when they are not. Most modern statisticians recommend always using Welch's version.

Effect Size: Cohen's d

The p-value tells you whether the difference is statistically significant. Cohen's d tells you how large it is in standard deviation units:

d = (x̄₁ - x̄₂) / s_p

Interpretation guidelines:

• d ≈ 0.2: small effect
• d ≈ 0.5: medium effect
• d ≈ 0.8: large effect

from scipy import stats
import numpy as np

pooled_std = np.sqrt(((len(control)-1)*control.var(ddof=1) + (len(treatment)-1)*treatment.var(ddof=1)) / (len(control) + len(treatment) - 2))
cohens_d = (treatment.mean() - control.mean()) / pooled_std
print(f"Cohen's d: {cohens_d:.3f}")

Common Pitfall: Comparing More Than Two Groups

If you have three or more groups, do not run multiple t-tests. Each test carries a 5% false positive rate, and the probability of at least one false positive across multiple tests grows rapidly. Use ANOVA instead, which we cover in the next topic.

Paired T-Test

The paired t-test compares means from the same group measured under two different conditions. It accounts for the correlation between paired observations, giving it more statistical power than a two-sample t-test when the pairing is meaningful.

When to Use It

Use a paired t-test when each observation in one group is naturally linked to a specific observation in the other group:

• Same user's behavior before and after a feature change
• Same product's performance on two different algorithms
• Same store's sales in two different months
• Matched pairs in a case-control study

The key is the natural pairing. If you can match each observation in group A to a specific observation in group B, the paired test is appropriate.

Why Pairing Increases Power

Consider measuring the effect of a new training program on employee productivity. You measure productivity before and after for each employee.

The paired t-test analyzes the differences (7, 3, 5, 5, 4). Because each person serves as their own control, the between-person variability (some people are naturally more productive) is eliminated. The test only sees the within-person change.

A two-sample t-test would treat the before and after measurements as independent groups, ignoring the pairing. This inflates the standard error because between-person variability adds noise. The paired test removes that noise, making it easier to detect a true effect.

The Test Statistic

The paired t-test is mathematically a one-sample t-test on the differences:

t = d̄ / (s_d / √n)

Where d̄ is the mean of the differences, s_d is the standard deviation of the differences, and n is the number of pairs.

from scipy import stats
import numpy as np

np.random.seed(42)
before = np.array([45, 38, 50, 42, 35, 48, 41, 39, 44, 37])
after = before + np.random.normal(4, 2, len(before))  # true effect of +4

t_stat, p_value = stats.ttest_rel(after, before)
print(f"Mean difference: {(after - before).mean():.2f}")
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

When Pairing Is Wrong

Do not use a paired test when the observations are not naturally paired. If you measure 100 users in group A and 100 different users in group B, there is no pairing. Forcing a paired test (e.g., by sorting both groups and pairing by rank) produces invalid results because the pairing is artificial.

The paired test assumes the differences are normally distributed. Check this assumption, especially with small sample sizes:

differences = after - before
stat, p = stats.shapiro(differences)
print(f"Shapiro-Wilk on differences: p = {p:.4f}")

A/B Testing with User-Level Metrics

In product analytics, paired tests are common when you measure the same users over time. For example, if you want to know whether a new feature increases weekly active usage, you compare each user's activity before and after the feature launch.

However, in a standard A/B test where users are randomly assigned to control or treatment, the two-sample t-test is correct because the groups are independent. Using a paired test on independent groups is a mistake that can produce artificially small p-values.

Choosing the Right T-Test

Selecting the correct t-test depends on your experimental design and data structure. The wrong choice can inflate false positives or waste statistical power.

Decision Framework

How many groups?
├── 1 group
│   └── Compare to a known value?
│       ├── Yes → One-sample t-test
│       └── No → descriptive statistics only
└── 2 groups
    ├── Are observations paired?
    │   ├── Yes → Paired t-test
    │   └── No → Two-sample t-test
    │       ├── Equal variances? → Student's t-test
    │       └── Unequal variances? → Welch's t-test
└── 3+ groups
    └── ANOVA (next topic)

Practical Comparison

from scipy import stats
import numpy as np

np.random.seed(42)

# Scenario 1: One-sample
sample = np.random.normal(103, 12, 80)
t1, p1 = stats.ttest_1samp(sample, 100)
print(f"One-sample: t={t1:.3f}, p={p1:.4f}")

# Scenario 2: Two-sample (independent)
group_a = np.random.normal(100, 15, 150)
group_b = np.random.normal(104, 15, 150)
t2, p2 = stats.ttest_ind(group_a, group_b, equal_var=False)
print(f"Two-sample (Welch): t={t2:.3f}, p={p2:.4f}")

# Scenario 3: Paired
before = np.random.normal(100, 15, 100)
after = before + np.random.normal(3, 5, 100)
t3, p3 = stats.ttest_rel(after, before)
print(f"Paired: t={t3:.3f}, p={p3:.4f}")

Sample Size Considerations

The t-test is robust to non-normality with moderate sample sizes (n > 30 per group) due to the Central Limit Theorem. With small samples:

• Check normality rigorously (Shapiro-Wilk, Q-Q plots)
• Consider non-parametric alternatives (Mann-Whitney U, Wilcoxon signed-rank)
• Be cautious with outliers, which have disproportionate influence on small samples

Non-Parametric Alternatives

When assumptions are severely violated:

from scipy import stats

# Mann-Whitney U (independent groups)
u_stat, p_value = stats.mannwhitneyu(group_a, group_b, alternative='two-sided')

# Wilcoxon signed-rank (paired)
w_stat, p_value = stats.wilcoxon(after, before)

Non-parametric tests make fewer assumptions but are less powerful when the parametric assumptions hold. They also test slightly different hypotheses (distributions rather than means), so interpret results accordingly.

Reporting T-Test Results

A complete report includes:

Test: Welch's two-sample t-test
Group A: mean = 100.2, sd = 14.8, n = 150
Group B: mean = 104.1, sd = 15.3, n = 150
Difference: 3.9 (95% CI: [0.4, 7.4])
t(297.3) = 2.71, p = 0.007
Cohen's d = 0.26 (small effect)

Report the test type, group statistics, effect size with confidence interval, test statistic with degrees of freedom, p-value, and effect size measure. This gives readers everything needed to evaluate the result.