Sampling Bias and Representativeness

What Selection Bias Is

Selection bias occurs when the process that determines which units enter your sample is correlated with the variable you are measuring. The sample is not a distorted version of the population in a random way; it is distorted in a specific direction that systematically pushes your estimate away from the truth.

The critical distinction: selection bias is not about sample size. A sample of 10 million users can be more biased than a sample of 100 if the selection mechanism is flawed. Bias is a property of the sampling process, not the sample size.

The Mechanism

Selection bias arises from three common mechanisms:

1. Self-selection: Units choose whether to participate. Users who voluntarily complete a satisfaction survey are systematically different from those who do not. They are often more extreme in their opinions, either very satisfied or very dissatisfied.

2. Conditioning on a collider: You filter the data on a variable that is caused by both the exposure and the outcome. Example: you analyze only users who made a purchase, then ask whether the recommendation algorithm influenced their decision. The act of purchasing is caused by both the recommendation and the user's pre-existing intent, so the relationship you estimate is distorted.

3. Survivorship in the frame: The sampling frame only contains units that survived some prior process. Analyzing only active users, only completed transactions, or only successful deployments introduces bias because the excluded units differ systematically from the included ones.

A Concrete Product Example

A product team runs an A/B test on a new onboarding flow. The metric is day-7 retention. The analysis includes only users who completed onboarding. But the new flow causes 15% of users to drop out during onboarding, compared to 5% in the control. By conditioning on completion, you exclude the users the new flow failed to retain, and you overestimate its effectiveness.

The correct denominator is all users who entered onboarding, not just those who finished it. This is an intention-to-treat principle: analyze users based on the group they were assigned to, not the treatment they actually received.

Detecting Selection Bias

You cannot detect selection bias from the sample alone. You need external information about the population or the selection process.

Diagnostic checks:

• Compare sample demographics to known population benchmarks (age, region, device type)
• Examine the distribution of the metric in the sample vs a trusted external source
• Look for implausible patterns: zero variance in a variable that should vary, or a correlation structure that contradicts domain knowledge
• Track the funnel: at each step where units are lost, ask whether the loss is random or systematic

The Classic Case

During World War II, statistician Abraham Wald was asked to reinforce bomber planes. The military wanted to add armor to the areas that showed the most bullet holes in returning aircraft. Wald pointed out the error: the planes that returned were the ones that survived. The areas with no bullet holes on returning planes were the most critical, because planes hit there never came back.

This is survivorship bias: you analyze only the units that passed through a selection process, and you ignore the ones that were filtered out. The survivors are systematically different from the full population, and the difference is exactly what you need to understand.

Survivorship Bias in Product Analytics

Churn analysis: You analyze the features of users who stayed for 12 months to understand what drives retention. But the users who churned at month 2 are invisible. The features you identify as predictive of retention may simply be correlated with early churn, and you would not know it.

App store reviews: You analyze 5-star reviews to understand what users love. But users who had a bad experience and uninstalled the app without leaving a review are excluded. Your analysis over-represents satisfied users.

A/B test winners: You analyze the experiments that showed statistically significant positive results. But the experiments that showed no effect or negative results were deprioritized and never published. Your internal experiment dashboard overestimates the true effect size of your team's interventions.

Quantifying the Bias

import pandas as pd
import numpy as np

# Simulated: true effect of a feature on retention
np.random.seed(42)
n = 10000
true_effect = 0.05  # 5 percentage point increase in retention

population = pd.DataFrame({
    'user_id': range(n),
    'feature': np.random.binomial(1, 0.5, n),
    'retention_prob': 0.30 + true_effect * np.random.binomial(1, 0.5, n)
})
population['retained'] = np.random.binomial(1, population['retention_prob'])

# Naive analysis: only look at retained users
naive = population[population['retained'] == 1]
print(f"Feature rate among retained: {naive['feature'].mean():.3f}")
print(f"Feature rate in population:  {population['feature'].mean():.3f}")

# Correct analysis: compare retention rates by feature
correct = population.groupby('feature')['retention'].mean()
print(f"Retention by feature:\n{correct}")

The naive analysis conditions on retention (a collider), which distorts the relationship between the feature and retention. The correct analysis compares retention rates across feature groups.

Mitigation Strategies

1. Define the denominator explicitly: Before any analysis, write down the exact population you are making a claim about. Then verify that your sample represents that population.

2. Track the full funnel: Document every step where units are lost. If you cannot recover the lost units, at least quantify how many were lost and hypothesize how they differ.

3. Use inverse probability weighting: If you can model the probability of survival, you can weight the survivors to represent the full population. This requires assumptions, but explicit assumptions are better than implicit ones.

4. Seek out the missing data: Sometimes the excluded units are available in a different table or system. Churned users still exist in the billing system. Uninstalled users still appear in acquisition logs.

The Problem

Non-response bias occurs when units that are sampled but do not differ systematically from units that do respond. It is a special case of selection bias where the selection mechanism is the decision to participate.

The bias is not caused by non-response alone. It is caused by non-response that is correlated with the variable of interest. If dissatisfied users are less likely to respond to a satisfaction survey, your estimate of mean satisfaction is biased upward. If non-response is random, you lose precision but not accuracy.

Why It Is Dangerous

Non-response bias is insidious because it is invisible in the data you have. You cannot compare responders to non-responders on the survey variable because you never observed it. You can only compare them on auxiliary variables that were collected before the survey.

Example: You send a survey to 10,000 users. 1,000 respond. You compare responders and non-responders on tenure, activity level, and plan type. They look similar. You conclude the sample is representative. But the survey asks about a new feature that only power users have tried, and power users are more likely to respond to surveys in general. Your estimate of feature satisfaction is biased, and your auxiliary checks did not catch it.

Diagnosing Non-Response

import pandas as pd
import numpy as np

# Simulated survey data
np.random.seed(42)
n = 10000
population = pd.DataFrame({
    'user_id': range(n),
    'activity_level': np.random.exponential(1.0, n),
    'satisfaction': np.random.normal(3.5, 1.0, n)
})

# Non-response is correlated with satisfaction (dissatisfied users skip the survey)
response_prob = 1 / (1 + np.exp(-(population['satisfaction'] - 3.0)))
population['responded'] = np.random.binomial(1, response_prob)

responders = population[population['responded'] == 1]
print(f"Mean satisfaction (responders): {responders['satisfaction'].mean():.2f}")
print(f"Mean satisfaction (population): {population['satisfaction'].mean():.2f}")
print(f"Response rate: {population['responded'].mean():.2f}")

The response rate alone tells you nothing about bias. You need to model the relationship between response propensity and the outcome variable.

Mitigation Strategies

1. Increase response rate: Follow-up reminders, shorter surveys, incentives. A higher response rate reduces the opportunity for bias, though it does not eliminate it.

2. Compare early vs late responders: Late responders are similar to non-responders. If early and late responders differ on the outcome variable, non-response bias is likely.

3. Use administrative data: If you have behavioral data on both responders and non-responders, you can weight the responders to match the population on those variables. This is post-stratification.

4. Sensitivity analysis: Assume the worst case. What would the non-responders' answers need to be to change your conclusion? If the conclusion is fragile, report it as fragile.

5. Report non-response rates transparently: A 10% response rate with unknown bias is a limitation, not a detail. State it explicitly in any analysis that uses survey data.

What Representativeness Means

A sample is representative if the distribution of relevant characteristics in the sample matches the distribution in the population. Representativeness is not a binary property. A sample can be representative on some dimensions and badly unrepresentative on others.

The relevant dimensions are those that affect the outcome variable. If you are estimating average session length, representativeness on device type and network quality matters. Representativeness on favorite color does not.

The Bias-Variance Tradeoff in Sampling Design

There is a tension between reducing bias and reducing variance:

• Probability sampling (SRS, stratified, cluster) gives you known selection probabilities, which allows unbiased estimation and valid uncertainty quantification. But it can be expensive and slow.

• Non-probability sampling (convenience samples, snowball samples, opt-in panels) is fast and cheap, but you cannot compute valid confidence intervals because the selection probabilities are unknown.

In analytical work, you often receive non-probability samples. Your job is to assess how far the sample deviates from the population on observable dimensions and to qualify your conclusions accordingly.

A Practical Framework

When you receive a dataset and need to assess representativeness:

1. Define the population: Write down exactly who the results should generalize to.

2. Identify benchmark distributions: Find external data on the population's characteristics (census data, platform analytics, industry reports).

3. Compare distributions: For each relevant characteristic, compare the sample distribution to the benchmark. Use a chi-square test for categorical variables and a Kolmogorov-Smirnov test for continuous variables.

4. Quantify the impact: For each mismatch, estimate how much it could bias your metric. Sensitivity analysis is more useful than a formal test here.

5. Weight or restrict: If the mismatch is large and the characteristic affects the outcome, apply post-stratification weights or restrict the analysis to the well-represented subpopulation.

Python: Comparing Sample to Population

import pandas as pd
import numpy as np
from scipy import stats

# Sample and population distributions of device type
sample_device = pd.Series(['mobile']*700 + ['desktop']*250 + ['tablet']*50)
population_device = pd.Series(['mobile']*5500 + ['desktop']*3500 + ['tablet']*1000)

# Chi-square test for distribution difference
sample_counts = sample_device.value_counts().sort_index()
pop_props = population_device.value_counts(normalize=True).sort_index()
expected = pop_props * len(sample_device)

chi2, p_value = stats.chisquare(sample_counts, expected)
print(f"Chi-square: {chi2:.2f}, p-value: {p_value:.2f}")

A significant result tells you the distributions differ. It does not tell you whether the difference matters for your specific analysis. That requires domain knowledge about how device type affects your metric.