Expectation and Variance

What Expected Value Means

The expected value of a random variable is the long-run average you would observe if you could repeat the experiment infinitely many times. It is not the most likely outcome. It is not the outcome you should expect on any single trial. It is the probability-weighted average of all possible outcomes.

For a discrete random variable X with outcomes x_i and probabilities p_i:

E[X] = Σ x_i * p_i

For a continuous random variable with probability density function f(x):

E[X] = ∫ x * f(x) dx

The expected value is a property of the distribution, not of any single observation. It tells you where the distribution is centered.

A Product Metric Example: Revenue Per User

Suppose a freemium product has three user types:

• Free users (90% of users): $0 revenue
• Basic subscribers (8%): $10/month
• Premium subscribers (2%): $50/month

The expected revenue per user is:

E[revenue] = 0.90 * 0 + 0.08 * 10 + 0.02 * 50 = 0 + 0.80 + 1.00 = $1.80

This does not mean each user generates $1.80. It means that across a large population, the average revenue per user converges to $1.80. The distribution is highly skewed — most users generate nothing, a few generate a lot — but the expectation captures the average behavior.

Computing Expected Value in Python

import numpy as np

# Discrete distribution
outcomes = np.array([0, 10, 50])
probabilities = np.array([0.90, 0.08, 0.02])

expected_value = (outcomes * probabilities).sum()
print(f'E[revenue] = ${expected_value:.2f}')
# Output: E[revenue] = $1.80

The element-wise multiplication weights each outcome by its probability, and the sum aggregates them. This is the definition of expected value applied directly.

Why Expectation Matters for Decision-Making

Expected value is the correct basis for comparing decisions under uncertainty. If you can choose between:

• Option A: guaranteed $1,000
• Option B: 50% chance of $2,500, 50% chance of $0

E[A] = $1,000. E[B] = 0.5 * 2500 + 0.5 * 0 = $1,250.

Option B has a higher expected value. If you can repeat this choice many times, B is better in the long run. For a single choice, B is riskier — you might get nothing. Expected value tells you the long-run optimal strategy, not the safe single-trial strategy.

This distinction is critical in product decisions. A feature that adds $0.10 in expected revenue per user but has high variance (some users love it, some hate it) might be the right bet if you have millions of users. The same feature with 100 users might produce a wildly different observed average due to variance.

Expectation vs. Median: When They Diverge

For symmetric distributions, the mean and median coincide. For skewed distributions, they diverge — and the divergence carries information.

Revenue per user is right-skewed: the mean ($1.80) exceeds the median ($0, since 90% of users pay nothing). Reporting the mean alone overstates what a typical user generates. Reporting the median alone understates the total revenue potential.

The rule: report both. The mean tells you about total revenue (mean * users). The median tells you about the typical user. The gap between them tells you about inequality in monetization.

Linearity of Expectation

The most powerful property of expectation is linearity:

E[aX + bY] = a*E[X] + b*E[Y]

This holds regardless of whether X and Y are independent. You do not need to know the joint distribution. You only need the marginal expectations.

This makes expectation uniquely tractable. Variance does not have this property (it requires knowing covariance), which is why expectation is the first tool you reach for when decomposing complex metrics.

Example: total revenue = revenue from new users + revenue from existing users. Even if these are correlated, E[total] = E[new] + E[existing]. You can decompose the expectation without understanding the dependence structure.

What Variance Measures

Expected value tells you where a distribution is centered. Variance tells you how spread out it is. A metric with high variance is unpredictable: individual observations can deviate far from the mean. A metric with low variance is stable: observations cluster tightly around the mean.

Var(X) = E[(X - E[X])²] = E[X²] - (E[X])²

The first form is the definition: the expected squared deviation from the mean. The second form is computationally convenient: the expected value of the square minus the square of the expected value.

Standard deviation is the square root of variance:

σ = sqrt(Var(X))

Standard deviation is in the same units as the original variable, which makes it more interpretable. If revenue per user has a mean of $1.80 and a standard deviation of $8.50, the large standard deviation relative to the mean signals high variability.

Computing Variance in Python

import numpy as np

outcomes = np.array([0, 10, 50])
probabilities = np.array([0.90, 0.08, 0.02])

mean = (outcomes * probabilities).sum()
variance = (probabilities * (outcomes - mean) ** 2).sum()
std = np.sqrt(variance)

print(f'Mean: ${mean:.2f}, Std: ${std:.2f}')
# Output: Mean: $1.80, Std: $8.50

The variance weights each squared deviation by its probability. Outcomes far from the mean contribute disproportionately because of the squaring — this is why variance is sensitive to outliers.

Variance of a Sum: Why Dependence Matters

For two random variables:

Var(X + Y) = Var(X) + Var(Y) + 2*Cov(X, Y)

The covariance term captures how X and Y move together. If they are independent, Cov(X, Y) = 0 and the variances add. If they are positively correlated, the variance of the sum exceeds the sum of variances. If they are negatively correlated, it is less.

This has direct implications for metric analysis. Suppose you decompose daily revenue into revenue from two product lines. If the product lines are positively correlated (both spike on holidays), the variance of total revenue is larger than you would expect from independent components. Risk assessment that ignores this correlation underestimates the true variability.

Variance and Sample Means: The Standard Error

When you compute a sample mean, you are estimating the population mean. The precision of that estimate depends on the variance of the underlying distribution and the sample size:

Var(sample_mean) = Var(X) / n
SE = σ / sqrt(n)

The standard error shrinks as the square root of the sample size. To halve the standard error, you need four times the data. This quadratic relationship is why increasing sample size has diminishing returns for precision.

Example: if the standard deviation of session duration is 120 seconds and you have 100 sessions, the standard error of the mean is 120 / 10 = 12 seconds. With 400 sessions, it is 120 / 20 = 6 seconds. With 10,000 sessions, it is 120 / 100 = 1.2 seconds.

Using Variance for Sample Sizing

Before running an experiment, you need to know how many samples are required to detect a meaningful effect. The sample size formula for comparing two means is:

n = (z_α/2 + z_β)² * 2σ² / δ²

Where:

• σ is the standard deviation of the metric
• δ is the minimum detectable effect
• z_α/2 is the critical value for the significance level
• z_β is the critical value for the desired power

The key insight: sample size scales with variance and inversely with the square of the effect size. A high-variance metric requires proportionally more data. A small effect requires quadratically more data.

from scipy import stats
import numpy as np

sigma = 120  # standard deviation of session duration (seconds)
delta = 10   # minimum detectable effect (seconds)
alpha = 0.05
power = 0.80

z_alpha = stats.norm.ppf(1 - alpha / 2)
z_beta = stats.norm.ppf(power)

n = ((z_alpha + z_beta) ** 2 * 2 * sigma ** 2) / (delta ** 2)
print(f'Required sample size per variant: {int(np.ceil(n))}')

This calculation prevents the most common experimental error: running an underpowered test that cannot detect the effect you care about.

Decomposing Metrics Using Linearity

Many product metrics are sums of components. Daily active users = new users + returning users. Revenue = transactions * average order value. When a metric is a sum, linearity of expectation lets you decompose its expected value:

E[DAU] = E[new users] + E[returning users]

This decomposition is exact and requires no independence assumption. It lets you attribute changes in the total to changes in the components. If DAU increased by 1,000 and new users increased by 600, then returning users must have increased by 400 (in expectation).

Variance of a Weighted Sum

When a metric is a weighted sum (e.g., revenue = Σ price_i * quantity_i), the variance depends on the weights and the covariances:

Var(Σ w_i X_i) = Σ w_i² Var(X_i) + Σ Σ w_i w_j Cov(X_i, X_j)

For independent components, the covariance terms vanish and the variance is the weighted sum of individual variances. For correlated components, you must estimate the covariance matrix — which is why correlated metrics are harder to analyze than independent ones.

Coefficient of Variation: Relative Variability

The coefficient of variation (CV) standardizes variance relative to the mean:

CV = σ / μ

A CV of 0.1 means the standard deviation is 10% of the mean — relatively stable. A CV of 2.0 means the standard deviation is twice the mean — extremely volatile.

CV is useful for comparing variability across metrics with different scales. Session duration (mean 300s, std 120s, CV 0.4) is less variable relative to its mean than revenue per user (mean $1.80, std $8.50, CV 4.7). The revenue metric is far noisier, which means you need much more data to detect changes in it.

Variance and Metric Monitoring

When you monitor a metric over time, you are watching a random process. The metric fluctuates around its expected value due to natural variance. The question is: how much fluctuation is normal?

Control charts answer this by plotting the metric over time with control bounds at ±2 or ±3 standard deviations from the mean. Points outside the bounds are statistically unlikely under the null hypothesis of no change, triggering investigation.

import pandas as pd
import numpy as np

df = pd.read_csv('daily_metrics.csv')  # columns: date, metric_value

mean = df['metric_value'].mean()
std = df['metric_value'].std()

df['upper_bound'] = mean + 2 * std
df['lower_bound'] = mean - 2 * std
df['anomaly'] = (df['metric_value'] > df['upper_bound']) | (df['metric_value'] < df['lower_bound'])

print(f'Anomalies detected: {df["anomaly"].sum()}')

The control bounds are derived from the variance. A metric with high variance has wide bounds and fewer false alarms but misses subtle changes. A metric with low variance has narrow bounds and detects small changes but produces more false alarms. The variance of the metric determines the sensitivity of your monitoring.

Variance Reduction Techniques

When a metric is too noisy to analyze effectively, you can reduce its variance:

1. Stratification. Split the population into homogeneous subgroups and analyze each separately. Within-group variance is lower than overall variance.

2. CUPED (Controlled-experiment Using Pre-Experiment Data). Use pre-experiment covariates to adjust the metric, removing variance explained by those covariates. This can reduce variance by 50% or more.

3. Aggregation. Average over longer time windows. The variance of a 7-day average is 1/7 of the daily variance (for independent days).

4. Transformation. Apply a variance-stabilizing transformation (e.g., log for multiplicative metrics). This makes the variance more uniform across the range of the metric.

Each technique trades something for variance reduction. Stratification requires defining meaningful strata. CUPED requires pre-experiment data. Aggregation reduces temporal resolution. Transformation complicates interpretation. Choose based on what you can afford to lose.

Expected Value Is Not the Whole Story

Two investments can have the same expected value but very different risk profiles:

• Investment A: 100% chance of $1,000 (variance = 0)
• Investment B: 50% chance of $2,500, 50% chance of $0 (variance = 1,562,500)

Both have E = $1,000... wait, E[B] = $1,250. Let me correct: Investment B has E = $1,250 with high variance. A risk-neutral decision-maker prefers B. A risk-averse decision-maker might prefer A despite the lower expected value.

This is why variance matters for decisions. Expected value tells you the average outcome. Variance tells you how much you might deviate from that average. For decisions where downside is costly (outages, churn spikes, budget overruns), variance is as important as expectation.

The Variance of A/B Test Outcomes

In an A/B test, the observed difference between variants is a random variable. Its expected value is the true effect. Its variance determines how precisely you can estimate that effect.

Var(difference) = Var(A)/n_A + Var(B)/n_B

If both variants have variance σ² and equal sample sizes n:

Var(difference) = 2σ²/n
SE(difference) = σ * sqrt(2/n)

The standard error of the difference shrinks with sample size. This is the mathematical foundation of sample sizing: you choose n to make the standard error small enough that the minimum detectable effect is statistically distinguishable from zero.

Variance and Metric Selection

When choosing between metrics to optimize, consider both sensitivity and variance:

• High-sensitivity, low-variance metrics (e.g., click-through rate on a high-traffic page) are ideal for experimentation. You can detect small effects with modest sample sizes.

• Low-sensitivity, high-variance metrics (e.g., annual revenue per user) are poor experimental metrics. You need enormous samples to detect meaningful changes.

• Proxy metrics (e.g., time on page as a proxy for engagement) reduce variance but introduce validity risk. The proxy might move without the true metric moving.

The standard approach: use a high-sensitivity, low-variance metric for the experiment, and validate that improvements in the proxy translate to improvements in the true metric over longer time horizons.

Quantifying Uncertainty in Reports

When you report a metric to stakeholders, report the uncertainty alongside the point estimate:

• "Conversion rate: 5.2% ± 0.3% (95% CI)"
• "Revenue per user: $1.80 ± $0.15 (95% CI)"

The confidence interval is derived from the standard error, which is derived from the variance. Without it, the point estimate is misleading — it implies a precision that the data does not support.

from scipy import stats
import numpy as np

samples = np.array([...])  # your metric observations
mean = samples.mean()
se = samples.std(ddof=1) / np.sqrt(len(samples))
ci = stats.t.interval(0.95, len(samples) - 1, loc=mean, scale=se)

print(f'Mean: {mean:.4f}, 95% CI: [{ci[0]:.4f}, {ci[1]:.4f}]')

The t-interval accounts for the uncertainty in the estimated standard deviation, which matters for small samples. For large samples, the normal approximation is sufficient.

The Bias-Variance Tradeoff in Metric Models

When you build a model to predict a metric (e.g., churn prediction), the prediction error decomposes into bias and variance:\n

MSE = Bias² + Variance + Irreducible Error

• Bias is the error from wrong assumptions. A model that assumes linear relationships when the truth is nonlinear has high bias.

• Variance is the error from sensitivity to the training data. A complex model that fits the training data perfectly but generalizes poorly has high variance.

The tradeoff: reducing bias (by making the model more complex) increases variance. Reducing variance (by simplifying the model) increases bias. The optimal model balances the two.

This tradeoff appears in metric analysis when you choose between simple and complex aggregations. A simple average has low variance but high bias if the population is heterogeneous. A segmented analysis has lower bias but higher variance because each segment has fewer observations. The right choice depends on the segment sizes and the heterogeneity of the population.