Experiment Readout

The Anatomy of an Experiment Readout

An experiment readout is the document that communicates what you learned and what you recommend. It is the deliverable that determines whether your experiment has impact. A well-structured readout drives decisions. A poorly structured readout creates confusion and erodes trust in the experimentation program.

Why Readout Structure Matters

Most experiment readouts fail not because the analysis is wrong, but because the communication is unclear. Stakeholders do not understand what was measured, what the result means, or what decision they should make. They see a p-value and a percentage and form their own interpretation — which is often wrong.

A good readout does three things:

1. States the recommendation clearly: Ship, no-ship, or iterate. The reader should not have to search for the conclusion.

2. Supports the recommendation with evidence: Confidence intervals, segment analysis, and guardrail checks that justify the decision.

3. Acknowledges limitations: What the experiment does not tell you, what risks remain, and what should be monitored after launch.

The Standard Readout Structure

A standard experiment readout contains these sections:

1. Summary: One paragraph stating the hypothesis, the result, and the recommendation. This is the only section some stakeholders will read. It must be self-contained.

2. Experiment design: Brief recap of the hypothesis, primary metric, sample size, and duration. This provides context for interpreting the result.

3. Results: The primary metric result with confidence interval, secondary metrics, and guardrail checks. This is the evidence.

4. Segment analysis: How the effect varies across user segments. This reveals heterogeneity and identifies risks.

5. Recommendation: Ship, no-ship, or iterate, with justification. This is the decision.

6. Limitations and risks: What the experiment does not capture, what could go wrong after launch, and what to monitor.

Writing the Summary

The summary is the most important section. It must answer three questions in three sentences:

1. What did you test?
2. What did you find?
3. What do you recommend?

Example:

We tested whether a new one-page checkout flow increases purchase conversion. The treatment group showed a 2.3% relative increase in conversion (95% CI: [1.1%, 3.5%], p = 0.001), with no degradation in guardrail metrics. We recommend shipping.

This summary is complete, specific, and actionable. A stakeholder can understand the result and the recommendation in 15 seconds.

Common Readout Mistakes

• Burying the recommendation: Making the reader hunt through pages of analysis to find the conclusion.
• Reporting only the point estimate: Saying "conversion increased by 2.3%" without a confidence interval. The point estimate is not the result — the confidence interval is.
• Ignoring guardrails: Reporting the primary metric result without checking whether guardrails were breached.
• Overstating certainty: Using language like "proves" or "confirms" when the data only supports a probabilistic conclusion.
• Hiding limitations: Not mentioning what the experiment does not capture, which leads to overconfident decisions.

Confidence Intervals and Effect Sizes

The confidence interval is the most important number in your experiment readout. It communicates both the magnitude of the effect and the uncertainty around that estimate. A readout that reports only a p-value or a point estimate is incomplete.

Why Confidence Intervals Matter

A point estimate says "conversion increased by 2.3%." A confidence interval says "conversion increased by 2.3%, and we are 95% confident the true effect is between 1.1% and 3.5%."

The confidence interval tells you:

1. The magnitude of the effect: How large is the improvement?
2. The precision of the estimate: How certain are we about the magnitude?
3. The practical significance: Is the effect large enough to matter?

A result can be statistically significant but practically meaningless. If the 95% CI is [0.1%, 0.3%], the effect is real but tiny. Whether it is worth shipping depends on the business context, not the p-value.

Calculating Confidence Intervals

For a two-proportion z-test, the confidence interval for the difference in proportions is:

CI = (p₂ - p₁) ± z_{α/2} × SE

Where the standard error is:

SE = sqrt(p₁(1-p₁)/n₁ + p₂(1-p₂)/n₂)

import numpy as np
from scipy.stats import norm

def proportion_ci(control_successes: int, control_n: int,
                  treatment_successes: int, treatment_n: int,
                  confidence: float = 0.95) -> dict:
    p1 = control_successes / control_n
    p2 = treatment_successes / treatment_n
    diff = p2 - p1
    
    se = np.sqrt(p1 * (1 - p1) / control_n + p2 * (1 - p2) / treatment_n)
    z = norm.ppf(1 - (1 - confidence) / 2)
    
    ci_lower = diff - z * se
    ci_upper = diff + z * se
    
    return {
        "control_rate": p1,
        "treatment_rate": p2,
        "absolute_diff": diff,
        "relative_diff": diff / p1,
        "ci_lower": ci_lower,
        "ci_upper": ci_upper,
        "ci_width": ci_upper - ci_lower
    }

result = proportion_ci(control_successes=950, control_n=10000,
                       treatment_successes=1180, treatment_n=10000)
print(f"Control: {result['control_rate']:.3f}")
print(f"Treatment: {result['treatment_rate']:.3f}")
print(f"Absolute diff: {result['absolute_diff']:.4f}")
print(f"Relative diff: {result['relative_diff']:.1%}")
print(f"95% CI: [{result['ci_lower']:.4f}, {result['ci_upper']:.4f}]")

Interpreting Confidence Intervals

When interpreting a confidence interval, ask three questions:

1. Does the interval include zero? If yes, the result is not statistically significant at the chosen confidence level.

2. Is the entire interval above the MDE? If yes, the effect is both statistically significant and practically meaningful.

3. Does the interval include values that would change the decision? If the interval includes both "worth shipping" and "not worth shipping" values, the experiment is inconclusive.

Relative vs. Absolute Effects

Always report both relative and absolute effects. A 50% relative increase sounds impressive, but if the baseline is 0.002%, the absolute increase is 0.001% — probably not worth shipping.

The second row has the largest relative effect but the smallest absolute effect. Which one matters depends on your business context.

Confidence Intervals for Ratios

For ratio metrics (e.g., revenue per user), the confidence interval is often calculated on the log scale and then exponentiated:

import numpy as np
from scipy.stats import norm

def ratio_ci(control_values: np.ndarray, treatment_values: np.ndarray,
              confidence: float = 0.95) -> dict:
    log_ratio = np.log(treatment_values.mean() / control_values.mean())
    se_log = np.sqrt(treatment_values.var() / len(treatment_values) +
                     control_values.var() / len(control_values)) / treatment_values.mean()
    z = norm.ppf(1 - (1 - confidence) / 2)
    
    ci_lower = np.exp(log_ratio - z * se_log)
    ci_upper = np.exp(log_ratio + z * se_log)
    
    return {
        "ratio": np.exp(log_ratio),
        "ci_lower": ci_lower,
        "ci_upper": ci_upper
    }

The log-scale approach ensures the confidence interval is always positive and symmetric on the log scale, which is more appropriate for ratio metrics.

Segment Analysis and Heterogeneity

Segment analysis examines whether the treatment effect varies across user segments. It reveals heterogeneity that the aggregate result hides and identifies risks that could cause post-launch problems.

Why Segment Analysis Matters

An experiment might show a positive aggregate result that is driven entirely by one segment. If you ship based on the aggregate, you might harm other segments.

Consider a new onboarding flow that increases activation by 3% overall. Segment analysis reveals:

• New users: +8% activation
• Returning users: -2% activation

The aggregate result is positive, but the treatment harms returning users. If returning users are a significant fraction of your user base, the net effect might be negative.

Choosing Segments

Segment on variables that are:

1. Measured before the experiment: Segments defined by pre-experiment behavior (e.g., historical purchase frequency) are valid. Segments defined by post-experiment behavior are contaminated by the treatment.

2. Theoretically relevant: Segment on variables that could plausibly moderate the treatment effect. There should be a hypothesis for why the effect might differ.

3. Large enough: Each segment must have enough users to produce a reliable estimate. A segment with 100 users will have a very wide confidence interval.

Common segmentation dimensions:

• User type (new vs. returning)
• Platform (iOS vs. Android vs. web)
• Geography
• Device type
• Historical behavior (power user vs. casual user)

Analyzing Segments

For each segment, calculate the treatment effect and confidence interval:

import pandas as pd
import numpy as np
from scipy.stats import norm

def segment_analysis(df: pd.DataFrame, segment_col: str,
                    outcome_col: str, group_col: str) -> pd.DataFrame:
    results = []
    for segment in df[segment_col].unique():
        seg_df = df[df[segment_col] == segment]
        control = seg_df[seg_df[group_col] == 'control'][outcome_col]
        treatment = seg_df[seg_df[group_col] == 'treatment'][outcome_col]
        
        p1 = control.mean()
        p2 = treatment.mean()
        diff = p2 - p1
        se = np.sqrt(p1 * (1 - p1) / len(control) + p2 * (1 - p2) / len(treatment))
        z = diff / se if se > 0 else 0
        p_value = 2 * (1 - norm.cdf(abs(z)))
        
        results.append({
            'segment': segment,
            'control_n': len(control),
            'treatment_n': len(treatment),
            'control_rate': p1,
            'treatment_rate': p2,
            'absolute_diff': diff,
            'relative_diff': diff / p1 if p1 > 0 else np.nan,
            'p_value': p_value,
            'significant': p_value < 0.05
        })
    
    return pd.DataFrame(results)

Multiple Comparisons in Segment Analysis

Segment analysis involves multiple comparisons. If you test 10 segments, you expect one to show a "significant" result by chance alone at α = 0.05. You must adjust for this.

Options:

1. Bonferroni correction: Divide α by the number of segments. Conservative but simple.

2. Benjamini-Hochberg: Control the false discovery rate. Less conservative, more powerful.

3. Hierarchical testing: Test the overall effect first. Only test segments if the overall effect is significant.

4. Interaction tests: Instead of testing each segment separately, test whether the treatment effect differs significantly between segments. This is more powerful than separate tests.

Interpreting Segment Results

When interpreting segment results, look for:

1. Consistent direction: Does the effect go in the same direction across all segments? If some segments show positive effects and others show negative effects, the aggregate result may be misleading.

2. Consistent magnitude: Are the effect sizes similar across segments? Large differences suggest heterogeneity that warrants investigation.

3. Actionable segments: Are there segments where the effect is particularly large or particularly small? These represent opportunities or risks.

When Segment Analysis Changes the Decision

Segment analysis should change your recommendation when:

• The aggregate result is positive but a significant segment shows a negative effect
• The aggregate result is negative but a high-value segment shows a large positive effect
• The effect is concentrated in a segment that does not represent your target users

In these cases, the recommendation should reflect the segment-level reality, not the aggregate average.

Making the Ship/No-Ship Recommendation

The recommendation is the point of the experiment. Everything in the readout leads to a clear, defensible decision. This chapter provides a framework for making that decision.

The Decision Framework

The ship/no-ship decision should be based on three criteria:

1. Statistical significance: Is the effect real? The confidence interval should not include zero.

2. Practical significance: Is the effect large enough to matter? The confidence interval should be entirely above the MDE.

3. Guardrail safety: Are guardrail metrics within acceptable bounds? No guardrail should be breached.

All three criteria must be met to ship. If any criterion fails, the recommendation is no-ship or iterate.

Decision Rules

The Iterate Recommendation

Sometimes the right recommendation is neither ship nor no-ship but iterate. Iterate when:

• The effect is positive but not statistically significant (underpowered)
• The effect is significant but smaller than the MDE (real but not meaningful)
• The effect is positive overall but negative for an important segment
• The guardrail metrics show a concerning trend but do not breach the threshold

An iterate recommendation should specify what to change: larger sample size, different metric, modified treatment, or different segment targeting.

Writing the Recommendation

The recommendation should be specific and actionable:

Bad: "The results are promising. We recommend further analysis."

Good: "We recommend shipping the new checkout flow. The treatment increased conversion by 2.3% (95% CI: [1.1%, 3.5%]), exceeding our 2% MDE. All guardrail metrics passed. We recommend a full rollout with monitoring of page load time for 2 weeks post-launch."

The good recommendation states the decision, the evidence, and the next steps. It is actionable.

Communicating Uncertainty

Every experiment result has uncertainty. A good readout communicates this uncertainty honestly:

• Report confidence intervals, not just point estimates
• State the false positive rate (α) and what it means
• Acknowledge what the experiment does not capture (long-term effects, novelty effects, interaction with other changes)
• Specify what to monitor after launch and what would trigger a rollback

Post-Launch Monitoring

The experiment does not end at launch. Specify:

1. What to monitor: Key metrics that could be affected by the treatment
2. For how long: The monitoring period (typically 2-4 weeks)
3. What triggers a rollback: Specific thresholds that would cause you to revert the treatment

## Post-Launch Monitoring

### Metrics to Monitor
- Conversion rate (primary)
- Page load time (guardrail)
- Customer support tickets (guardrail)

### Monitoring Period
2 weeks post-launch

### Rollback Triggers
- Conversion drops below 9.5% for 3 consecutive days
- Page load time exceeds 2.5 seconds for 3 consecutive days
- Support tickets increase by >10% week-over-week

The Readout as a Living Document

An experiment readout is not a one-time document. It should be updated as new data becomes available. If the experiment is extended, update the results. If post-launch monitoring reveals problems, document them. The readout is the record of what you learned and what you decided — it should be complete and accurate.