Metrics and Guardrails

The Metric Hierarchy

Not all metrics are created equal. In an A/B test, metrics serve different roles, and confusing these roles leads to bad decisions. A metric hierarchy clarifies which metrics drive the decision, which provide context, and which protect against harm.

Primary Metrics

The primary metric is the single number that determines whether you ship the treatment. It is the metric your experiment is powered to detect a change in. You should have exactly one primary metric per experiment.

The primary metric must be:

• Sensitive: It can detect the effect you care about within your sample size and experiment duration.
• Robust: It is not easily gamed or manipulated by the treatment in a way that does not reflect real user value.
• Decision-relevant: A change in this metric maps directly to a business or product goal.

If you cannot name your primary metric in one sentence, your experiment is not ready to launch.

Secondary Metrics

Secondary metrics provide context for the primary metric result. They help you understand why the primary metric changed and whether the change is real or an artifact.

For example, if your primary metric is conversion rate, secondary metrics might include:

• Average order value (did conversion increase because more people bought, or because they bought cheaper items?)
• Time to purchase (did the new flow speed up checkout?)
• Bounce rate (did users engage more or just leave faster?)

Secondary metrics are not used for the ship/no-ship decision. They are used to build a narrative around the result and to generate hypotheses for future experiments.

Guardrail Metrics

Guardrail metrics are safety checks. They are metrics that must not degrade beyond a predefined threshold. If a guardrail metric breaches its threshold, you do not ship — regardless of how good the primary metric looks.

Guardrails protect against treatments that improve one metric by harming another. A new recommendation algorithm might increase click-through rate but also increase server load. If server load exceeds your threshold, the experiment is a failure, even if CTR improved.

Common guardrail categories:

Why You Need All Three

A common mistake is to track only the primary metric. This is like driving a car while looking only at the speedometer. You might be going the right speed, but you are about to run out of gas.

The metric hierarchy works together:

1. Primary metric tells you if the treatment works.
2. Secondary metrics tell you why it works.
3. Guardrail metrics tell you if it is safe.

Without guardrails, you ship treatments that improve a narrow metric while degrading the overall user experience. Without secondary metrics, you ship treatments you do not understand, which makes iteration difficult.

Choosing Metrics That Measure What Matters

The hardest part of experiment design is not statistics. It is choosing metrics that reflect real user value rather than convenient proxies.

The Proxy Problem

Most metrics are proxies for the outcome you actually care about. You want to increase user satisfaction, but you cannot measure satisfaction directly, so you measure session duration. You want to improve product quality, but you cannot measure quality directly, so you measure error rate.

Proxy metrics are useful but dangerous. They are useful because they are measurable. They are dangerous because they can diverge from the true outcome you care about.

Consider a team that optimizes for session duration. They add more content to each page, which increases time on site. But users are not more satisfied — they are just taking longer to find what they need. The proxy metric improved while the true outcome degraded.

Criteria for Good Metrics

A good experiment metric should satisfy these criteria:

1. Actionable: A change in the metric implies a specific action. If the metric improves, you ship. If it degrades, you iterate.

2. Sensitive: The metric can detect the effect size you care about within your experimental constraints. A metric that changes by 0.001% when you make a meaningful change is not sensitive enough.

3. Robust: The metric is not easily gamed. If the treatment can improve the metric without improving user experience, it is not robust.

4. Timely: The metric can be measured within your experiment duration. If you need six months to see a change, you cannot run a two-week experiment.

5. Interpretable: A change in the metric has a clear, unambiguous meaning. If stakeholders need a 30-minute meeting to interpret the result, the metric is too complex.

Metric Types and Their Tradeoffs

Behavioral metrics (clicks, views, purchases) are directly observable and easy to measure. They are sensitive to short-term changes. But they can be gamed — a misleading headline increases clicks without increasing value.

Outcome metrics (revenue, retention, lifetime value) reflect real business value. They are robust but slow. You may need weeks or months to detect a change.

Perceptual metrics (survey responses, NPS) capture user sentiment. They are leading indicators of behavior but noisy and subject to response bias.

The best experiments use a combination: a primary outcome metric, secondary behavioral metrics for context, and guardrail metrics for safety.

Building a Metric Tree

A metric tree decomposes a high-level goal into measurable components. For example:

Goal: Increase user engagement
├── Primary: Weekly active users
├── Secondary:
│   ├── Sessions per user per week
│   ├── Actions per session
│   └── Feature adoption rate
└── Guardrails:
    ├── Session duration (must not decrease)
    ├── Error rate (must not increase)
    └── Support tickets (must not increase)

The metric tree ensures that your experiment metrics are connected to business goals. It also reveals tradeoffs — if you increase sessions per user but decrease session duration, is that good or bad?

Practical Metric Selection

When selecting metrics for an experiment:

1. Start with the decision you need to make. What would make you ship or not ship?
2. Identify the metric that best captures that decision. This is your primary metric.
3. Identify metrics that could be harmed by the treatment. These are your guardrails.
4. Identify metrics that would help you understand the result. These are your secondary metrics.
5. Verify that your primary metric is sensitive enough to detect the effect size you care about. If not, you need a larger sample size or a more sensitive metric.

Metric Gaming and Goodhart's Law

Goodhart's law states: "When a measure becomes a target, it ceases to be a good measure." In the context of A/B testing, this means that any metric you optimize for will eventually be gamed — either intentionally by the treatment or unintentionally by the system's response to the treatment.

How Metric Gaming Happens

Metric gaming occurs when a treatment improves a metric without improving the underlying user value the metric is supposed to represent. Common patterns:

Vanity metrics: The treatment increases a metric that looks good but does not correlate with business outcomes. A new pop-up increases email signups, but the signups are from users who never engage with the product.

Short-term optimization: The treatment improves a short-term metric while degrading long-term value. A aggressive discount increases purchases this week but trains users to wait for discounts.

Selection effects: The treatment changes the composition of users who trigger the metric. A new onboarding flow increases activation rate, but only because it filters out low-intent users before they reach the activation step.

Threshold effects: The treatment pushes users just past a metric threshold without meaningful improvement. A progress bar shows 99% completion, which increases completion rate, but users do not actually finish the task.

Detecting Metric Gaming

Metric gaming is difficult to detect because the metric genuinely improves. You need to look beyond the primary metric:

1. Check guardrail metrics. If the primary metric improves but guardrails degrade, the improvement may be an artifact.

2. Segment the result. If the improvement is concentrated in a segment that does not represent your target users, it may not be real.

3. Look at downstream metrics. If the primary metric improves but downstream metrics (retention, revenue) do not, the improvement may not be meaningful.

4. Run a holdout. Keep a small percentage of users on the control for an extended period. If the effect decays over time, it was a short-term artifact.

Defending Against Metric Gaming

The best defense against metric gaming is a well-designed metric hierarchy:

• Use outcome metrics as primary metrics whenever possible. Revenue, retention, and lifetime value are harder to game than clicks and views.

• Set guardrail metrics that capture the negative effects of gaming. If you optimize for clicks, guardrail with conversion rate. If you optimize for conversion, guardrail with refund rate.

• Pre-register your metrics. If you commit to a primary metric before the experiment, you cannot switch to a different metric after seeing the result.

• Use composite metrics that combine multiple signals. A single metric is easy to game; a weighted combination is harder.

Goodhart's Law in Practice

A classic example: a team optimizes for "number of support tickets resolved." Agents start closing tickets quickly without resolving the underlying issue. The metric improves, but customer satisfaction drops and tickets reopen at a higher rate.

The fix is not to abandon the metric but to pair it with a guardrail. Track both "tickets resolved" and "ticket reopen rate." If resolution rate improves but reopen rate also improves, the optimization is gaming the metric.

The Long-Term View

Metric gaming is not always malicious. Often, it is an emergent property of a system optimizing for a narrow objective. The treatment designer did not intend to game the metric — they just found a way to improve it that does not align with user value.

This is why experiment design requires thinking about the system, not just the metric. Ask: "If I optimize for this metric, what is the easiest way the system could improve it without improving user value?" Then design guardrails that detect that failure mode.

Building a Metric Selection Framework

A metric selection framework is a structured process for choosing experiment metrics. It ensures consistency across experiments and prevents ad-hoc metric selection that leads to gaming and confusion.

Step 1: Define the Decision

Every experiment exists to inform a decision. Start by writing down the decision in one sentence:

• "Should we ship the new checkout flow?"
• "Should we change the pricing page layout?"
• "Should we launch the new recommendation algorithm?"

The decision determines the metrics. If the decision is about shipping a feature, your primary metric should measure the feature's impact on user behavior or business outcomes.

Step 2: Identify the User Value Proposition

What user problem does the treatment solve? How would you know if it solved it? The answer to these questions points to your primary metric.

If the treatment reduces friction in checkout, the user value is faster, easier checkout. The primary metric should measure whether users complete checkout more often (conversion rate) or more quickly (time to purchase).

Step 3: Map Metrics to the User Journey

Map the user journey and identify metrics at each stage:

Awareness → Consideration → Conversion → Retention → Advocacy
   ↓              ↓              ↓           ↓           ↓
Impressions   Product views   Purchase    Return rate  NPS
CTR           Add to cart     AOV         Churn rate   Referrals

Your primary metric should be at the stage where the treatment has its primary effect. If the treatment is a new product page, the primary metric is likely at the consideration stage (product views, add to cart).

Step 4: Identify Risks and Guardrails

For each metric, ask: "How could the treatment harm this metric?" The answers become your guardrail metrics.

Step 5: Validate Sensitivity

Before launching, verify that your primary metric can detect the effect size you care about. Use historical data to estimate the metric's variance, then calculate the minimum detectable effect (MDE) for your planned sample size.

from scipy.stats import norm
import math

def minimum_detectable_effect(baseline_rate: float, sample_size_per_group: int,
                                alpha: float = 0.05, power: float = 0.80) -> float:
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)
    p1 = baseline_rate
    p2 = baseline_rate  # under null
    pooled_se = math.sqrt(2 * p1 * (1 - p1) / sample_size_per_group)
    mde = (z_alpha + z_beta) * pooled_se
    return mde

mde = minimum_detectable_effect(baseline_rate=0.10, sample_size_per_group=50000)
print(f"MDE: {mde:.4f} ({mde/0.10*100:.2f}% relative)")

If the MDE is larger than the effect you care about, you need a larger sample size or a more sensitive metric.

Step 6: Document and Socialize

Write the metric selection into your pre-registration document. Share it with stakeholders before the experiment launches. This creates accountability and prevents post-hoc metric changes.

A metric selection framework is not bureaucracy. It is a forcing function that makes you think clearly about what you are measuring and why. The time you spend selecting metrics before the experiment saves you from invalid results after it.