Hypothesis Tests and Confidence Intervals

State the question before calculating

A hypothesis test evaluates how compatible observed data are with a specified null model. Define the null hypothesis, the alternative, the outcome, the sampling unit, and the analysis plan before inspecting the result. The null might state that two variants have equal mean conversion, while the alternative states a difference or a direction of change. The choice between one-sided and two-sided tests must follow the decision question and be made before seeing which direction the data favors.

A test statistic summarizes how far the observed result is from what the null predicts, in units of its sampling uncertainty. The p-value is the probability, assuming the null model and its assumptions, of observing a result at least as extreme as the one measured. It is not the probability that the null is true, the probability the result happened by chance, or the size of the effect. A small p-value indicates incompatibility with the null under the model; it does not establish a useful or causal effect.

Choose a significance threshold alpha before analysis if a threshold-based decision is needed. A Type I error rejects a true null; a Type II error fails to detect a specified effect. Statistical power is the probability of detecting that effect under the alternative. Power depends on sample size, variability, effect size, test design, and alpha. Increasing sample size can detect a tiny effect that has no product value, so define a minimum practically important effect as well.

Interpret confidence intervals

A confidence interval is produced by a procedure that, over repeated samples under its assumptions, covers the true parameter at the stated rate. A 95 percent confidence procedure does not mean there is a 95 percent probability that this one fixed parameter lies inside the interval after the data are observed. In ordinary reporting, the interval communicates an estimate and its sampling uncertainty across plausible effect sizes.

The interval carries information that a thresholded p-value hides. A narrow interval around a small improvement may show precision but little practical value. A wide interval spanning meaningful harm and meaningful benefit means the evidence is inconclusive, even if a point estimate looks favorable. For a two-sided test using the same model and assumptions, an interval excluding the null value corresponds to rejection at the matching significance level. This relationship is useful, but neither result removes model assumptions or bias in the data collection.

Check assumptions and design

Inference relies on the design and model. Independent observations, a suitable randomization scheme, correct standard errors, and a defined outcome matter. Repeated measurements from one person are not independent rows. Clustered assignment requires cluster-aware uncertainty. Skewed outcomes, unequal variances, censoring, and missingness may require a different estimator or resampling procedure. A t-test should not be selected merely because the outcome is numeric.

Predefine the primary metric and analysis window. Repeatedly checking a p-value and stopping as soon as it crosses a threshold inflates false positives unless a sequential method accounts for the monitoring. Selecting a favorable segment after looking at many cuts creates a similar problem. Exploratory findings can generate hypotheses, but confirmatory claims need a planned analysis or independent replication.

Account for multiple comparisons

Testing many metrics, variants, or subgroups increases the chance of at least one false positive. Bonferroni controls family-wise error by dividing alpha across tests, which can be conservative. False-discovery-rate procedures control the expected fraction of false discoveries among reported findings and can be more suitable for broad screening. The correction should reflect which claims will be treated as confirmatory and how the results will be used.

Report effect size, interval, sample size, assumptions, and practical threshold alongside the p-value. A test is evidence about a defined estimand under a design, not a universal truth detector. Strong statistical practice combines uncertainty with domain consequences and transparent decisions.

Plan an A/B experiment

Suppose a product team compares a new onboarding flow with the current one. Define the randomization unit, such as user or account, and avoid assigning correlated members of one account to different variants if they influence each other. Choose one primary outcome and an analysis window. State the null, alternative, and minimum practically important effect before looking at results. Check that the assignment mechanism is actually random and that outcome capture is consistent between variants.

Estimate sample size using the expected baseline rate, variability, target effect, alpha, and desired power. The estimate is conditional on those assumptions; it is not a guarantee. If traffic is lower than expected, the study may end with a wide interval that includes both no effect and a meaningful benefit. Decide whether to extend the study, accept uncertainty, or gather more evidence. Do not repeatedly peek and stop at the first favorable p-value unless using a valid sequential design.

Estimate and report the effect

Calculate the difference in the primary outcome and a confidence interval. Report the absolute difference as well as relative lift so stakeholders can understand scale. If the interval includes no effect but excludes a practically important harm or benefit, describe what conclusions are and are not supported. A non-significant test does not prove equivalence; equivalence requires a prespecified margin and an appropriate test.

Check whether the analysis matches the randomization. If users are assigned by account, estimate uncertainty at the account level or use a method that accounts for clustering. If conversion is binary, use a method suitable for proportions. If the outcome is heavily skewed, inspect robust or bootstrap intervals and retain the original units. Do not choose the method after seeing which one creates a significant result.

Handle many metrics and segments

Teams often inspect retention, clicks, revenue, latency, and several user segments. Choose a primary metric and label the rest exploratory or adjust for multiplicity. Bonferroni is simple but may reduce power across many correlated outcomes. False-discovery-rate control is often useful for screening, while a small set of confirmatory outcomes may need family-wise control. Record the testing family so the adjustment is auditable.

Segment analysis should be motivated by the product question. A result that appears only in one small group may be noise, especially when many groups were tested. Show sample sizes and intervals, and replicate the finding before making a high-impact change. Avoid converting a post-hoc subgroup into a permanent targeting policy without a new confirmatory design.

Write a decision, not just a test result

A clear experiment report states the population, assignment unit, period, primary outcome, estimated effect, uncertainty interval, test assumptions, and decision threshold. Explain whether the result is statistically distinguishable from the null and whether its magnitude is practically useful. If the interval remains broad, identify the information needed for a decision rather than forcing a binary conclusion.

As an exercise, imagine a two-percentage-point lift with a 95 percent interval from minus one to plus five points. The estimate is positive, but the evidence remains compatible with a small harm and a useful improvement. The right conclusion depends on the cost of deployment and the value of more data, not on the sign of the point estimate alone. This is the difference between statistical calculation and decision analysis.

Do not turn thresholds into truth

A p-value just below 0.05 is not categorically different from one just above it. The threshold is a convention for a decision process, not a boundary between true and false. Report the effect and interval, and explain whether the range includes outcomes that would change the decision. Large datasets can make negligible effects statistically detectable; small datasets can leave important effects uncertain.

Power analysis depends on a target effect and assumptions about variability. A study powered only for a very large effect may miss a smaller but still valuable one. A post-hoc power calculation based on the observed effect adds little beyond the confidence interval. Plan sample size before collecting data and report when assumptions were wrong. If an experiment is underpowered, describe the uncertainty rather than claiming no effect.

Match method to the decision

Multiple testing corrections reduce false discoveries but can reduce power. Use a correction that matches the claim family and decision cost. For broad exploratory model screening, false-discovery-rate control may be more appropriate than strict family-wise control. For a small number of safety-critical outcomes, stronger family-wise protection may be justified. Make the distinction between confirmatory and exploratory analysis explicit.

When the goal is to show that a change is not meaningfully worse, a conventional null test that fails to reject a difference is insufficient. Define a non-inferiority or equivalence margin before analysis and use a procedure designed for that claim. When assumptions for a parametric test fail, consider robust methods, permutation tests, or bootstrap intervals, while respecting randomization and dependence structure.

Design and reporting pitfalls

Randomization does not fix noncompliance, missing outcomes, interference between participants, or an unrepresentative sample. Identify the estimand: intention-to-treat, treatment-on-treated, or another effect. Clustered and repeated measurements require appropriate uncertainty. Data leakage can make an ML experiment appear more certain than it is if feature engineering or model selection uses the test outcomes.

For model comparisons, use paired evaluation when predictions are made on the same examples, and account for sampling variability across examples or time periods. Do not treat every row or every random seed as an independent replicate if they share data. Pre-register the metric and evaluation split when possible, and keep a final untouched test set for the final claim.

In an interview, define the null and alternative, explain the p-value conditionally, and distinguish it from effect size and practical value. Interpret a confidence interval using repeated-sampling language. Discuss power, multiple comparisons, and a failure mode such as peeking or post-hoc subgroup selection. Finish by connecting the result to a decision threshold and study design. This demonstrates statistical reasoning rather than a ritual search for significance.

For example, an estimated conversion lift of one percentage point with a 95 percent interval from minus half a point to two and a half points does not support a precise claim of benefit. The range includes a small harm and a useful gain. A team should compare that uncertainty with deployment cost, user impact, and the value of collecting more data. State the decision threshold before looking at the interval, and do not translate a confidence level into a posterior probability for this one parameter.