ANOVA for Multiple Groups

Why ANOVA Instead of Multiple T-Tests

When comparing means across three or more groups, the temptation is to run pairwise t-tests. This approach is wrong, and understanding why is essential to grasping what ANOVA actually does.

The Multiple Comparisons Problem

Suppose you have four groups (A, B, C, D) and you run all six pairwise t-tests. Each test at α = 0.05 has a 5% chance of a false positive. The probability of at least one false positive across all six tests is:

P(at least one false positive) = 1 - (1 - 0.05)^6 = 1 - 0.735 = 0.265

A 26.5% false positive rate is unacceptable. This is the multiple comparisons problem: the more tests you run, the more likely you are to find a "significant" result by chance alone.

You could apply a Bonferroni correction (dividing α by the number of tests), but this dramatically reduces power. With six tests, each would need p < 0.0083 to be significant. Real effects get missed.

What ANOVA Does Differently

ANOVA (Analysis of Variance) tests all groups simultaneously with a single test. It asks: "Is there any difference among the group means?" rather than "Is group A different from group B?"

H₀: μ_A = μ_B = μ_C = μ_D     (all group means are equal)
H₁: At least one mean differs    (not all means are equal)

This single test maintains the nominal α level. If the overall test is significant, you then use post-hoc tests (with appropriate corrections) to identify which specific groups differ.

The Variance Decomposition

ANOVA's name comes from its approach: it analyzes variance. The total variability in the data is split into two components:

Total variance = Between-group variance + Within-group variance

• Between-group variance measures how much the group means differ from the grand mean. Large between-group variance suggests the groups are genuinely different.
• Within-group variance measures how much individual observations vary within each group. This is the noise or error term.

The F-statistic is the ratio:

F = Between-group variance / Within-group variance

If the groups truly differ, between-group variance will be large relative to within-group variance, producing a large F-statistic. If the groups are identical, both variance components estimate the same underlying noise, and F will be close to 1.

A Concrete Example

You test three onboarding flows and measure completion rates:

Flow A: [28, 32, 30, 29, 31, 27, 33, 30, 28, 31]  mean = 30.0
Flow B: [35, 38, 36, 34, 37, 35, 39, 36, 35, 37]  mean = 36.2
Flow C: [31, 33, 32, 30, 34, 31, 35, 32, 31, 33]  mean = 32.2

The grand mean is 32.8. Between-group variance captures how far each group mean (30.0, 36.2, 32.8) is from 32.8. Within-group variance captures the spread of individual observations around their group means.

from scipy import stats
import numpy as np

flow_a = np.array([28, 32, 30, 29, 31, 27, 33, 30, 28, 31])
flow_b = np.array([35, 38, 36, 34, 37, 35, 39, 36, 35, 37])
flow_c = np.array([31, 33, 32, 30, 34, 31, 35, 32, 31, 33])

f_stat, p_value = stats.f_oneway(flow_a, flow_b, flow_c)
print(f"F-statistic: {f_stat:.3f}")
print(f"p-value: {p_value:.6f}")

The large F-statistic and tiny p-value indicate that at least one flow differs from the others. Post-hoc tests will identify which ones.

The F-Statistic and ANOVA Table

The F-statistic follows an F-distribution under the null hypothesis. The F-distribution has two parameters: degrees of freedom for the numerator (between groups) and degrees of freedom for the denominator (within groups).

Degrees of Freedom

For one-way ANOVA with k groups and N total observations:

Between-group df = k - 1
Within-group df = N - k
Total df = N - 1

With 3 groups and 30 total observations: between-group df = 2, within-group df = 27.

The ANOVA Table

The results are typically presented in an ANOVA table:

Source          | SS    | df | MS    | F     | p
----------------|-------|----|-------|-------|------
Between groups  | 192.4 | 2  | 96.2  | 45.8  | <0.001
Within groups   | 56.7  | 27 | 2.1   |       |
Total           | 249.1 | 29 |       |       |

Where:

• SS (Sum of Squares) measures total variation
• df (degrees of freedom) counts independent pieces of information
• MS (Mean Square) = SS / df, the variance estimate
• F = MS_between / MS_within

Computing ANOVA Manually

import numpy as np
from scipy import stats

groups = [flow_a, flow_b, flow_c]
k = len(groups)
N = sum(len(g) for g in groups)
grand_mean = np.mean(np.concatenate(groups))

# Between-group sum of squares
ss_between = sum(len(g) * (g.mean() - grand_mean)**2 for g in groups)

# Within-group sum of squares
ss_within = sum(((g - g.mean())**2).sum() for g in groups)

# Mean squares
ms_between = ss_between / (k - 1)
ms_within = ss_within / (N - k)

# F-statistic
f_stat = ms_between / ms_within
p_value = 1 - stats.f.cdf(f_stat, k-1, N-k)

print(f"F({k-1}, {N-k}) = {f_stat:.3f}, p = {p_value:.6f}")

Interpreting the F-Statistic

The F-statistic is a ratio of variances. Under H₀, both variance components estimate the same population variance, so F ≈ 1. Under H₁, between-group variance includes the true effect plus noise, making F > 1.

The p-value comes from the F-distribution. Large F-values are in the right tail, which is the rejection region. The F-distribution is asymmetric (bounded at 0, long right tail), so there is only one rejection region.

Effect Size: Eta-Squared

ANOVA's effect size is eta-squared (η²), the proportion of total variance explained by group membership:

η² = SS_between / SS_total

eta_squared = ss_between / (ss_between + ss_within)
print(f"η² = {eta_squared:.3f}")

Interpretation:

• η² ≈ 0.01: small effect
• η² ≈ 0.06: medium effect
• η² ≈ 0.14: large effect

A significant F-test with a small η² means the groups differ statistically but the differences may not be practically important. Always report η² alongside the p-value.

One-Way vs. Two-Way ANOVA

One-way ANOVA has one independent variable (factor) with multiple levels. Two-way ANOVA has two factors and can test interactions between them.

For example, if you test both the button color (green, blue) and the page layout (compact, spacious), a two-way ANOVA can detect:

• Main effect of color
• Main effect of layout
• Interaction effect (color works differently depending on layout)

Interaction effects are often the most interesting findings in product experiments. A feature might help on desktop but hurt on mobile. One-way ANOVA would average these effects and miss the pattern.

Post-Hoc Tests

A significant ANOVA result tells you that at least one group differs, but not which ones. Post-hoc tests (Latin for "after this") identify the specific pairs that differ while controlling the overall false positive rate.

Why Not Just Run T-Tests?

Running uncorrected pairwise t-tests after a significant ANOVA still inflates the false positive rate. Post-hoc tests adjust the p-values or critical values to account for the number of comparisons.

Tukey's HSD (Honestly Significant Difference)

Tukey's HSD is the most common post-hoc test for all pairwise comparisons. It controls the family-wise error rate (FWER) — the probability of making at least one false positive across all comparisons.

from scipy import stats
from statsmodels.stats.multicomp import pairwise_tukeyhsd
import numpy as np

# Combine data with group labels
all_data = np.concatenate([flow_a, flow_b, flow_c])
groups = ['A']*len(flow_a) + ['B']*len(flow_b) + ['C']*len(flow_c)

tukey = pairwise_tukeyhsd(all_data, groups, alpha=0.05)
print(tukey)

Output:

Multiple Comparison of Means - Tukey HSD
===================================================
 group1 group2  meandiff  p-adj   lower   upper  reject
---------------------------------------------------
     A      B     6.20    0.001   3.42    8.98   True
     A      C     2.20    0.041   0.04    4.36   True
     B      C    -4.00    0.008  -6.58   -1.42   True
---------------------------------------------------

The p-adj column shows adjusted p-values. The reject column indicates whether the difference is significant after correction.

Bonferroni Correction

The Bonferroni correction divides α by the number of comparisons:

α_adjusted = α / m

Where m is the number of pairwise comparisons. For 3 groups, m = 3, so α_adjusted = 0.05/3 ≈ 0.017.

from scipy import stats
import numpy as np
from itertools import combinations

group_dict = {'A': flow_a, 'B': flow_b, 'C': flow_c}
pairs = list(combinations(group_dict.keys(), 2))

for g1, g2 in pairs:
    t_stat, p_value = stats.ttest_ind(group_dict[g1], group_dict[g2])
    p_adjusted = min(p_value * len(pairs), 1.0)  # Bonferroni
    print(f"{g1} vs {g2}: p = {p_value:.4f}, p-adj = {p_adjusted:.4f}")

Bonferroni is simple but conservative. It controls FWER strictly, which means it can miss real effects (low power). Use it when false positives are very costly.

Dunnett's Test

When comparing multiple treatment groups against a single control, Dunnett's test is more powerful than Tukey's because it only makes k-1 comparisons instead of k(k-1)/2.

from statsmodels.stats.multicomp import MultiComparison

mc = MultiComparison(all_data, groups)
result = mc.allpairtest(stats.ttest_ind, method='bonferroni')
print(result[0])

Choosing a Post-Hoc Test

Interpreting Post-Hoc Results

Report the post-hoc results alongside the ANOVA:

ANOVA: F(2, 27) = 45.8, p < 0.001, η² = 0.77

Post-hoc (Tukey's HSD, α = 0.05):
- Flow B > Flow A: diff = 6.2, p-adj = 0.001
- Flow B > Flow C: diff = 4.0, p-adj = 0.008
- Flow C > Flow A: diff = 2.2, p-adj = 0.041

All three flows differ significantly. Flow B performs best. The effect size (η² = 0.77) indicates that flow type explains 77% of the variance in completion rates — a very large effect.

Assumptions and Violations

ANOVA relies on the same assumptions as the t-test, plus one additional requirement. Violations can distort p-values and lead to incorrect conclusions.

The Assumptions

1. Independence. Observations within and across groups are independent. Violated by repeated measures, clustered data, or temporal dependencies.

2. Normality. The residuals (observations minus group means) are approximately normally distributed. ANOVA is robust to mild violations with balanced designs and moderate sample sizes.

3. Homogeneity of variances (homoscedasticity). All groups have approximately equal population variances. This is the assumption most likely to be violated in practice.

Checking Assumptions

from scipy import stats
import numpy as np

# Normality of residuals
all_data = np.concatenate([flow_a, flow_b, flow_c])
group_means = np.repeat([g.mean() for g in [flow_a, flow_b, flow_c]], [len(flow_a), len(flow_b), len(flow_c)])
residuals = all_data - group_means

stat, p_norm = stats.shapiro(residuals)
print(f"Shapiro-Wilk (residuals): p = {p_norm:.4f}")

# Homogeneity of variances
stat, p_var = stats.levene(flow_a, flow_b, flow_c)
print(f"Levene's test: p = {p_var:.4f}")

Handling Violated Assumptions

Non-normality:

• With large samples (n > 30 per group), ANOVA is robust due to the CLT
• For small samples with skewness, use the Kruskal-Wallis test (non-parametric alternative)
• Transform the data (log, square root) to reduce skewness

Unequal variances:

• Use Welch's ANOVA, which does not assume equal variances
• Apply a variance-stabilizing transformation
• Use the Brown-Forsythe test

# Kruskal-Wallis (non-parametric ANOVA)
h_stat, p_value = stats.kruskal(flow_a, flow_b, flow_c)
print(f"Kruskal-Wallis: H = {h_stat:.3f}, p = {p_value:.6f}")

Welch's ANOVA

Standard ANOVA pools variances across groups. When group variances differ substantially, this pooling produces an inaccurate F-statistic. Welch's ANOVA uses weighted variances and adjusts degrees of freedom.

import pingouin as pg
import pandas as pd

df = pd.DataFrame({
    'value': np.concatenate([flow_a, flow_b, flow_c]),
    'group': ['A']*len(flow_a) + ['B']*len(flow_b) + ['C']*len(flow_c)
})

result = pg.welch_anova(dv='value', between='group', data=df)
print(result)

Balanced vs. Unbalanced Designs

A balanced design has equal sample sizes in all groups. ANOVA is most robust when designs are balanced. With unbalanced designs:

• Type I error rates can inflate if variances are unequal
• Type II error rates increase
• Consider weighted least squares or mixed-effects models

Outliers

Outliers have a disproportionate influence on ANOVA because they inflate within-group variance. This reduces the F-statistic and can mask real effects.

Check for outliers visually (boxplots) and statistically (Grubbs' test, IQR method). If you find outliers:

• Verify they are data entry errors
• Consider robust methods (trimmed means, bootstrap ANOVA)
• Report results with and without outliers

A Complete ANOVA Workflow

from scipy import stats
import numpy as np

# 1. Check assumptions
# 2. Run ANOVA
f_stat, p_value = stats.f_oneway(flow_a, flow_b, flow_c)

# 3. If significant, run post-hoc tests
if p_value < 0.05:
    from statsmodels.stats.multicomp import pairwise_tukeyhsd
    all_data = np.concatenate([flow_a, flow_b, flow_c])
    groups = ['A']*len(flow_a) + ['B']*len(flow_b) + ['C']*len(flow_c)
    tukey = pairwise_tukeyhsd(all_data, groups, alpha=0.05)
    print(tukey)

# 4. Report effect size
# 5. Interpret results in context

Always follow this sequence. Running post-hoc tests when the overall ANOVA is not significant is a mistake — it inflates the false positive rate without the protection of the omnibus test.