Sampling Methods

The Problem Sampling Solves

You are an analyst at a streaming platform with 80 million active users. Product leadership wants to know whether a new recommendation algorithm increases watch time. You cannot instrument every user, store every event, or wait for a full census before making a decision. You need an answer that is fast enough to act on and accurate enough to trust.

Sampling is the discipline of selecting a subset of a population so that measurements on the subset generalize to the whole. The entire field of statistical inference rests on one idea: a well-chosen small group can tell you about a large group, with quantifiable uncertainty.

Population vs Sample

The population is the complete set of units you want to learn about. The sample is the subset you actually observe. The gap between them is where all sampling theory lives.

• Population parameter: the true value you want (e.g., mean watch time across all 80M users)
• Sample statistic: the value you compute from your sample (e.g., mean watch time of 10,000 users)
• Sampling error: the difference between the two, which you can reduce but never eliminate

The goal is not to eliminate sampling error. The goal is to make it small enough that decisions do not change, and to quantify how large it might be.

The Sampling Frame

Before choosing a method, you need a sampling frame: the list of all population units from which you will draw. The frame is where many real-world sampling failures originate. If your frame omits part of the population, no sampling method can recover those missing units.

Example: You want to survey all app users, but your frame only contains users who logged in during the past week. Users who churned last month are invisible to every method you apply. This is a coverage error, and it is a bias problem, not a variance problem.

Four Methods You Will Use

Each method is a different answer to the same question: given limited budget, which units do you observe?

Definition and Mechanics

Simple random sampling (SRS) gives every unit in the population an equal probability of selection, and every combination of n units has the same probability of being chosen. It is the baseline against which all other methods are compared.

The procedure is straightforward:

1. Number every unit in the frame from 1 to N
2. Generate n random numbers in that range without replacement
3. Select the corresponding units

Python Implementation

import pandas as pd
import numpy as np

# Population: 100,000 users with watch time in minutes
np.random.seed(42)
population = pd.DataFrame({
    'user_id': range(1, 100001),
    'watch_minutes': np.random.gamma(shape=2, scale=30, size=100000)
})

# Simple random sample of 1,000
sample = population.sample(n=1000, random_state=42)

print(f"Population mean: {population['watch_minutes'].mean():.2f}")
print(f"Sample mean:     {sample['watch_minutes'].mean():.2f}")
print(f"Standard error:  {sample['watch_minutes'].std() / np.sqrt(len(sample)):.2f}")

Why SRS Works

SRS is unbiased: the expected value of the sample mean equals the population mean. This is not a claim that any single sample is accurate. It is a claim that the procedure does not systematically over- or under-estimate.

The variance of the sample mean under SRS is:

Var(x̄) = (1 - n/N) * σ²/n

The term (1 - n/N) is the finite population correction. When the sample is a small fraction of the population, it is close to 1 and can be ignored. When you sample a large fraction, it reduces your variance because you have observed a meaningful portion of the population.

When SRS Fails

SRS ignores structure. Suppose 90% of your users are free-tier and 10% are premium, and premium users watch 5x more content. An SRS of 1,000 might, by chance, include only 70 premium users instead of the expected 100. That fluctuation directly moves your estimate of mean watch time.

SRS also requires a complete frame. If you cannot enumerate the population, you cannot give every unit an equal chance of selection. In practice, many analytical datasets are convenience samples that already violate SRS assumptions before you begin.

The Core Idea

Stratified sampling divides the population into non-overlapping groups called strata, then draws an independent random sample from each stratum. The strata are chosen so that units within a stratum are similar to each other and different from units in other strata.

The streaming example: stratify by subscription tier (free, premium, family). Within each tier, watch time is more similar than across tiers. By sampling each tier separately, you guarantee that every tier is represented in proportion to its size, eliminating the random fluctuation that SRS produces.

Proportional vs Discretionary Allocation

Proportional allocation samples each stratum in proportion to its population size. If premium users are 10% of the population, they are 10% of the sample. This preserves the population composition and is the default choice.

Discretionary (optimal) allocation samples more heavily from strata that are more variable or more expensive to measure. If premium users have 5x the variance of free users, you might sample them at a higher rate to reduce the overall variance of your estimate. The formula for optimal allocation is:

n_h = n * (N_h * σ_h) / Σ(N_i * σ_i)

where N_h is the stratum size and σ_h is the stratum standard deviation.

Python Implementation

# Stratified sampling by subscription tier
population['tier'] = np.where(
    population['user_id'] % 10 == 0, 'premium', 'free'
)

# Proportional allocation: 10% premium, 90% free
strata = population.groupby('tier', group_keys=False).apply(
    lambda x: x.sample(frac=0.01, random_state=42)
)

print(strata.groupby('tier')['watch_minutes'].agg(['mean', 'std', 'count']))

Why Stratification Reduces Variance

The variance of a stratified estimator is the weighted sum of within-stratum variances. Because you remove the between-stratum variation from the sampling error, the total variance is always less than or equal to what SRS would produce with the same sample size.

The gain is largest when strata are homogeneous within and heterogeneous between. If stratification does not match the actual structure of the data, the benefit shrinks toward zero.

When to Use Stratified Sampling

• You need reliable estimates for subgroups, not just the overall population
• The population has clear, measurable segments that differ in the metric of interest
• You have auxiliary information (tier, region, device type) available for the full population
• You want to guarantee representation of small but important groups

The cost: you need stratum membership for every unit in the frame, and you need to manage multiple sampling processes. For a one-time analysis, this overhead is rarely worth it. For recurring reporting, it pays for itself in reduced variance.

Cluster Sampling

Cluster sampling groups the population into clusters, randomly selects a subset of clusters, and observes every unit within the chosen clusters. It inverts the logic of stratified sampling: strata are homogeneous within, clusters are heterogeneous within (ideally each cluster is a miniature of the population).

Why use it? Because it is often the only feasible method. Suppose you want to survey users across 200 cities. Building a frame of every user in every city is expensive. But you can randomly select 10 cities and survey everyone in them. The cost savings are enormous.

The tradeoff is variance. Because units within a cluster tend to be similar (same city, same network, same regional promotions), each additional unit within a cluster provides less new information than a unit from a new cluster. The design effect quantifies this:

Design effect = 1 + (m - 1) * ICC

where m is the average cluster size and ICC is the intra-cluster correlation. A design effect of 3 means your cluster sample of 1,000 has the same variance as an SRS of 333. You need 3x the sample size to match SRS precision.

Systematic Sampling

Systematic sampling selects every k-th unit from an ordered frame after a random start. If you need 1,000 from 100,000, you pick a random number between 1 and 100, then take units at positions start, start+100, start+200, and so on.

It is simple to implement and spreads the sample evenly across the frame. It is widely used in manufacturing quality control and in database sampling where the frame is a sorted table.

The danger: if the frame has a periodic pattern that aligns with k, systematic sampling can produce a badly biased sample. Example: you sample every 7th day of user activity, and weekends have systematically different behavior. Your sample over-represents weekdays and under-represents weekends.

Python Implementation

# Cluster sampling: randomly select 5 of 50 clusters
population['cluster'] = population['user_id'] % 50
selected_clusters = np.random.choice(population['cluster'].unique(), size=5, replace=False)
cluster_sample = population[population['cluster'].isin(selected_clusters)]

# Systematic sampling: every 100th unit after random start
k = 100
start = np.random.randint(0, k)
systematic_sample = population.iloc[start::k]

Choosing Between Methods

The decision is driven by three factors:

1. Frame quality: Can you enumerate the population? If not, cluster sampling with geographic or temporal clusters is often the only option.
2. Cost per unit: If reaching scattered units is expensive, cluster sampling reduces cost by concentrating effort.
3. Variance tolerance: If you need tight estimates and the population has clear segments, stratified sampling is worth the overhead.

In analytical work, you rarely design the sampling process yourself. You inherit data that was sampled by a product team, a logging system, or a third-party provider. Your job is to recognize which method was used, what biases it introduces, and whether the resulting data can support the inference you need.