Bayes Reasoning for Metrics

The Problem Bayes Solves

You have a belief about the world (a prior). You observe new evidence. You want to update your belief. Bayes' theorem is the mathematically correct way to perform that update.

P(H | E) = P(E | H) * P(H) / P(E)

Where:

• P(H) is the prior: your belief about hypothesis H before seeing evidence E.
• P(E | H) is the likelihood: how probable the evidence is if H is true.
• P(H | E) is the posterior: your updated belief after seeing the evidence.
• P(E) is the marginal probability of the evidence, acting as a normalizer.

The denominator expands to:

P(E) = P(E | H) * P(H) + P(E | not H) * P(not H)

This is the law of total probability applied to the evidence. It ensures the posterior is a valid probability between 0 and 1.

Why the Posterior Differs from the Likelihood

The most common analytical error is treating P(E | H) as if it were P(H | E). They are reciprocals in a sense, but not numerically equal. The likelihood asks: if the hypothesis is true, how expected is this evidence? The posterior asks: given this evidence, how likely is the hypothesis?

The difference is the base rate. A test can be highly likely to produce a positive result when the condition is present (high sensitivity), yet still produce mostly false positives when the condition is rare. Bayes' theorem forces you to account for both.

A Worked Example: Conversion Rate Estimation

Suppose you are analyzing a new landing page. Historically, 5% of visitors convert (the prior). You run a new page for a day and observe 100 visitors, 8 of whom convert.

The naive estimate is 8%. But the prior says 5%. Which do you trust?

Bayesian updating combines them. Using a Beta prior (Beta(5, 95) to represent 5 conversions in 100 historical trials) and the new data (8 conversions in 100 trials), the posterior is Beta(13, 187):

from scipy import stats
import numpy as np

alpha_prior, beta_prior = 5, 95
conversions, trials = 8, 100

alpha_post = alpha_prior + conversions
beta_post = beta_prior + (trials - conversions)

posterior_mean = alpha_post / (alpha_post + beta_post)
print(f'Posterior mean conversion rate: {posterior_mean:.4f}')
# Output: 0.0650

The posterior mean of 6.5% is pulled toward the prior (5%) but moved by the data (8%). With more data, the posterior converges to the observed rate. With less data, it stays closer to the prior. This is the correct way to combine historical knowledge with new observations.

The Role of the Prior

The prior is not a bias to eliminate — it is information to incorporate. Ignoring the prior and using only the current sample throws away historical knowledge. The prior regularizes your estimate, preventing small samples from producing extreme conclusions.

The strength of the prior depends on how much data it represents. A prior built from 10,000 historical sessions should dominate a new sample of 100 sessions. A prior built from 100 sessions should not. The Beta distribution's parameters (alpha, beta) encode this: larger values mean a stronger prior.

When to Use Bayesian vs. Frequentist Approaches

Use Bayesian updating when:

• You have a meaningful prior (historical data, domain knowledge).
• Samples are small and you need regularization.
• You want a full distribution over the parameter, not just a point estimate.

Use frequentist methods when:

• You have no prior information.
• You need a simple, reproducible point estimate.
• The sample size is large enough that the prior becomes irrelevant.

In practice, most A/B tests use frequentist methods (t-tests, confidence intervals) because the sample sizes are large. Bayesian methods shine for low-traffic experiments, anomaly detection, and any setting where you are updating beliefs incrementally as data arrives.

The Setup

Your monitoring system watches a metric (e.g., error rate) and fires an alert when the metric crosses a threshold. The system has:

• Sensitivity (true positive rate): 95% — it catches 95% of real anomalies.
• Specificity: 95% — it stays silent 95% of the time when nothing is wrong.

True anomalies occur 1% of the time. The system fires an alert. What is the probability that a real anomaly is occurring?

Applying Bayes' Theorem

P(anomaly | alert) = P(alert | anomaly) * P(anomaly) / P(alert)

Compute the denominator:

P(alert) = P(alert | anomaly) * P(anomaly) + P(alert | no anomaly) * P(no anomaly)
          = 0.95 * 0.01 + 0.05 * 0.99
          = 0.0095 + 0.0495 = 0.059

Now the posterior:

P(anomaly | alert) = 0.95 * 0.01 / 0.059 = 0.161

Only 16% of alerts are real anomalies. 84% are false positives. This is the false positive paradox: a test that is 95% accurate produces mostly false alarms when the condition is rare.

Why This Happens

The base rate (1%) is so low that the false positives from the large healthy population (5% of 99% = 4.95%) overwhelm the true positives from the small anomalous population (95% of 1% = 0.95%). The numerator is small; the denominator is dominated by false positives.

This is not a flaw in the detector. It is a mathematical consequence of the base rate. No amount of detector improvement fixes it unless the improvement is asymmetric — increasing specificity (reducing false positives) matters far more than increasing sensitivity when the condition is rare.

Computing Alert Precision in Python

import numpy as np

# Simulate 100,000 time periods
np.random.seed(42)
n = 100_000
anomaly_rate = 0.01
sensitivity = 0.95
specificity = 0.95

# Ground truth
anomalies = np.random.random(n) < anomaly_rate

# Alerts: true positives + false positives
alerts = np.where(
    anomalies,
    np.random.random(n) < sensitivity,      # catch real anomalies
    np.random.random(n) < (1 - specificity)  # false alarms
)

true_positives = (alerts & anomalies).sum()
alert_precision = true_positives / alerts.sum()
print(f'Alert precision: {alert_precision:.3f}')
# Output: ~0.16

The simulation confirms the analytical result. Run it with different base rates to see how precision changes.

Improving Alert Precision

1. Raise the threshold. A higher threshold reduces false positives (increases specificity) at the cost of missing some real anomalies. For rare conditions, this trade is usually worth it.

2. Use a prior that reflects the base rate. If you know anomalies are rare, incorporate that into the alerting logic. A Bayesian alert system computes P(anomaly | evidence) and only alerts when the posterior exceeds a threshold.

3. Require multiple signals. If two independent detectors must both fire, the false positive rate drops multiplicatively (0.05 * 0.05 = 0.0025), while the true positive rate drops less severely (0.95 * 0.95 = 0.9025). The posterior improves dramatically.

4. Segment by context. An anomaly at 3 AM is more likely real than the same anomaly at 3 PM. Conditioning on context changes the base rate, which changes the posterior.

The Cost of False Positives

Alert fatigue is not just an inconvenience — it is a reliability engineering problem. When 84% of alerts are false, operators learn to ignore alerts. When they ignore alerts, real anomalies cause outages. The solution is not more alerts; it is better-calibrated alerts that respect the base rate.

The Frequentist A/B Test and Its Limits

A standard A/B test computes a p-value: the probability of observing a difference at least as large as the one measured, assuming there is no real difference (the null hypothesis). If p < 0.05, you reject the null and declare a winner.

The p-value answers a question nobody asked. It is P(data | no difference). Decision-makers want P(difference | data) — the probability that the variant is actually better given what we observed. These are not the same, and confusing them is the most common error in experiment analysis.

The Bayesian Alternative

A Bayesian A/B test computes the posterior distribution of the conversion rate for each variant, then computes the probability that one variant beats the other:

from scipy import stats
import numpy as np

# Variant A: 500 conversions in 10,000 sessions
# Variant B: 550 conversions in 10,000 sessions
alpha_a, beta_a = 500, 9500
alpha_b, beta_b = 550, 9450

# Sample from posterior distributions
n_samples = 100_000
samples_a = stats.beta.rvs(alpha_a, beta_a, size=n_samples)
samples_b = stats.beta.rvs(alpha_b, beta_b, size=n_samples)

prob_b_beats_a = (samples_b > samples_a).mean()
print(f'P(B > A | data) = {prob_b_beats_a:.3f}')
# Output: ~0.95

This is a direct answer to the decision question: there is a 95% probability that variant B converts better than variant A. No p-value, no null hypothesis, no arbitrary significance threshold.

Interpreting the Posterior Distribution

The posterior is not just a probability — it is a full distribution. You can compute:

• Expected loss: If you choose B and B is actually worse, how much conversion do you lose on average?

loss_if_b_worse = np.maximum(samples_a - samples_b, 0)
expected_loss = loss_if_b_worse.mean()
print(f'Expected loss if B is worse: {expected_loss:.5f}')

• Credible interval: The range where the true conversion rate lies with 95% probability.

ci_low, ci_high = stats.beta.ppf([0.025, 0.975], alpha_b, beta_b)
print(f'95% credible interval for B: [{ci_low:.4f}, {ci_high:.4f}]')

These quantities support decision-making in a way that p-values cannot. The expected loss tells you the cost of being wrong. The credible interval tells you the precision of your estimate.

Early Stopping Without Penalty

Frequentist tests require a fixed sample size. Peeking at results and stopping early inflates the false positive rate. Bayesian tests do not have this problem — you can check the posterior at any time and make a decision based on the current evidence.

This does not mean you should stop arbitrarily. It means the decision rule is explicit: stop when P(B > A | data) exceeds a threshold (e.g., 0.95) or when the expected loss falls below a tolerance (e.g., 0.1% conversion). The threshold encodes your risk tolerance, which is how decisions should be made.

When Bayesian and Frequentist Agree

With large samples and uninformative priors, the Bayesian posterior concentrates around the observed rate, and P(B > A | data) approaches 1 when the frequentist p-value approaches 0. The methods converge. They diverge when samples are small or when the prior is informative — exactly the settings where the Bayesian approach is most valuable.

Practical Recommendation

Use Bayesian methods for:

• Low-traffic experiments where the prior matters.
• Sequential monitoring where you check results incrementally.
• Decisions where you need to quantify the cost of being wrong.

Use frequentist methods for:

• Large-sample experiments where the prior is irrelevant.
• Situations requiring a simple, standardized decision rule.
• Regulatory or compliance contexts where p-values are expected.

The best analysts understand both and choose based on the decision context, not habit.

Where Priors Come From

A prior is not a guess — it is a probability distribution that encodes existing knowledge. Sources of priors include:

1. Historical data. Last quarter's conversion rate is a strong prior for this quarter's rate, adjusted for known changes.

2. Domain knowledge. An experienced product manager might say "redesigns typically move conversion by 2-5%." That statement can be encoded as a prior distribution.

3. Meta-analysis. If similar experiments have been run before, their results form a prior for the current experiment.

4. Uniform prior. When you truly have no information, use a uniform prior (Beta(1,1)). The data dominates completely.

Encoding a Domain Expert's Judgment

Suppose a product manager says "I expect the new onboarding flow to improve activation by 3 percentage points, but I could see it being anywhere from 1 to 5 points." You can encode this as a normal prior centered at 0.03 with a standard deviation of 0.01:

from scipy import stats
import numpy as np

prior_mean = 0.03
prior_std = 0.01

# Sample from the prior
prior_samples = stats.norm.rvs(prior_mean, prior_std, size=100_000)
print(f'Prior: mean={prior_samples.mean():.4f}, std={prior_samples.std():.4f}')

The prior is now a concrete distribution you can update with data. The key is translating qualitative judgments into quantitative distributions — a skill that improves with practice.

Sensitivity Analysis: How Much Does the Prior Matter?

A Bayesian analysis should always include a sensitivity analysis: re-run the analysis with different priors and check whether the conclusion changes. If the conclusion is robust across reasonable priors, the data is driving the result. If it flips, the prior is driving the result, and you need more data.

from scipy import stats
import numpy as np

def posterior_prob_b_wins(alpha_a, beta_a, alpha_b, beta_b, n=100_000):
    samples_a = stats.beta.rvs(alpha_a, beta_a, size=n)
    samples_b = stats.beta.rvs(alpha_b, beta_b, size=n)
    return (samples_b > samples_a).mean()

# Weak prior: Beta(1, 1) for both
print('Weak prior:', posterior_prob_b_wins(1 + 500, 1 + 9500, 1 + 550, 1 + 9450))

# Strong prior: Beta(50, 950) for both (centered at 5%)
print('Strong prior:', posterior_prob_b_wins(50 + 500, 950 + 9500, 50 + 550, 950 + 9450))

# Informative prior favoring A: Beta(100, 900) for A, Beta(1, 1) for B
print('Prior favoring A:', posterior_prob_b_wins(100 + 500, 900 + 9500, 1 + 550, 1 + 9450))

If P(B > A) stays above 0.9 across all three priors, the conclusion is robust. If it drops below 0.9 with the prior favoring A, the data is not strong enough to overcome a skeptical prior, and you should collect more data before deciding.

The Prior-Posterior Update as a Workflow

Bayesian analysis is iterative. The posterior from one experiment becomes the prior for the next. This creates a learning loop:

1. Start with a prior based on historical data or domain knowledge.
2. Run an experiment and compute the posterior.
3. Use the posterior as the prior for the next experiment.
4. Over time, the posterior concentrates around the true value as data accumulates.

This workflow is how mature experimentation programs operate. Each experiment builds on the last, and the prior ensures that knowledge is carried forward rather than discarded.

Common Mistakes in Prior Selection

Mistake 1: Using a point estimate as a prior. A prior is a distribution, not a single number. Using "5%" as a prior ignores uncertainty. Use Beta(5, 95) or a normal distribution centered at 0.05.

Mistake 2: Making the prior too strong. A prior built from 100,000 historical sessions will dominate a new experiment with 1,000 sessions. Scale the prior to the amount of historical data it represents.

Mistake 3: Ignoring the prior's influence. Always run a sensitivity analysis. If you cannot show that the conclusion is robust to the prior, you have not done your job.

Mistake 4: Confusing subjective with arbitrary. A prior based on domain expertise is subjective but informed. A prior pulled from thin air is arbitrary. The former is a feature of Bayesian analysis; the latter is a misuse.