Correlation and Covariance

What Covariance Measures

Covariance measures how two variables change together. If one variable tends to increase when the other increases, the covariance is positive. If one tends to decrease when the other increases, the covariance is negative. If there is no consistent pattern, the covariance is near zero.

Population Covariance: Cov(X, Y) = Σ(xᵢ − μₓ)(yᵢ − μᵧ) / N

Sample Covariance: cov(X, Y) = Σ(xᵢ − x̄)(yᵢ − ȳ) / (n − 1)

The formula multiplies the deviation of each x from its mean by the deviation of each y from its mean, then averages those products. When both deviations are positive (both above their means) or both negative (both below their means), the product is positive. When one is above and the other below, the product is negative.

import pandas as pd
import numpy as np

# Two variables that move together
df = pd.DataFrame({
    'ad_spend': [100, 200, 300, 400, 500],
    'sales': [150, 280, 410, 520, 650]
})

covariance = df['ad_spend'].cov(df['sales'])
print(f"Covariance: {covariance:.2f}")  # 27500.00

The Problem with Covariance

Covariance has a critical limitation: it is not bounded and its magnitude depends on the units of the variables. A covariance of 27,500 sounds large, but is it? If ad_spend is in dollars and sales is in dollars, the covariance is in dollars squared — a unit that has no intuitive meaning.

# Same relationship, different units
df2 = pd.DataFrame({
    'ad_spend_k': [0.1, 0.2, 0.3, 0.4, 0.5],  # in thousands
    'sales_k': [0.15, 0.28, 0.41, 0.52, 0.65]       # in thousands
})

covariance2 = df2['ad_spend_k'].cov(df2['sales_k'])
print(f"Covariance (thousands): {covariance2:.6f}")  # 0.027500

The relationship between the variables is identical. But the covariance changed from 27,500 to 0.0275 purely because of the unit change. This makes covariance impossible to interpret in isolation and impossible to compare across different pairs of variables.

When to Use Covariance

Despite its limitations, covariance is not useless:

1. It is the foundation of correlation. Pearson correlation is essentially normalized covariance.
2. It is used in portfolio theory. Covariance between asset returns is a key input in Modern Portfolio Theory for measuring diversification benefit.
3. It is used in multivariate analysis. The covariance matrix is the input for Principal Component Analysis (PCA) and many other multivariate techniques.

But for most descriptive analytics — understanding the relationship between two variables — you want correlation, not covariance.

Definition and Interpretation

The Pearson correlation coefficient (r) measures the strength and direction of a linear relationship between two variables. It is calculated by dividing the covariance by the product of the standard deviations:

r = Cov(X, Y) / (σₓ × σᵧ)

This normalization produces a value between −1 and +1:

• r = +1 — perfect positive linear relationship
• r = −1 — perfect negative linear relationship
• r = 0 — no linear relationship
• |r| > 0.7 — strong linear relationship
• 0.3 < |r| < 0.7 — moderate linear relationship
• |r| < 0.3 — weak linear relationship

import pandas as pd
import numpy as np

# Strong positive linear relationship
df = pd.DataFrame({
    'ad_spend': [100, 200, 300, 400, 500],
    'sales': [150, 280, 410, 520, 650]
})

r = df['ad_spend'].corr(df['sales'])
print(f"Pearson r: {r:.4f}")  # 0.9992

What Pearson Correlation Does and Does Not Capture

Pearson correlation only measures linear relationships. This is its greatest limitation.

import numpy as np
import pandas as pd

# Perfect non-linear (quadratic) relationship
x = np.linspace(-5, 5, 100)
y = x ** 2

df = pd.DataFrame({'x': x, 'y': y})
r = df['x'].corr(df['y'])
print(f"Pearson r for y = x²: {r:.4f}")  # ~0.0 (no linear relationship)

The relationship between x and y is perfect — y is exactly x squared. But the Pearson correlation is approximately zero because the relationship is not linear. If you only looked at the correlation coefficient, you would conclude there is no relationship, which is completely wrong.

Always plot the data before interpreting correlation. A scatter plot reveals non-linear patterns, outliers, and clusters that the correlation coefficient hides.

Pearson Correlation Assumptions

For Pearson correlation to be meaningful:

1. Both variables should be continuous. Pearson is designed for interval/ratio data.
2. The relationship should be linear. Non-linear relationships will have misleadingly low correlations.
3. Both variables should be roughly normally distributed. Extreme skew or outliers can distort the correlation.
4. The data should be homoscedastic. The spread of y should be roughly constant across all values of x.

Calculating Correlation for Multiple Variables

In practice, you often want the correlation between all pairs of numeric variables in a dataset:

import pandas as pd

df = pd.DataFrame({
    'ad_spend': [100, 200, 300, 400, 500],
    'sales': [150, 280, 410, 520, 650],
    'website_visits': [500, 800, 1200, 1500, 2000],
    'customer_satisfaction': [3.5, 3.8, 4.1, 4.3, 4.5]
})

corr_matrix = df.corr()
print(corr_matrix)

The correlation matrix shows the Pearson correlation between every pair of variables. This is the first thing you should compute when exploring a new dataset — it reveals which variables move together and which do not.

The Rank-Based Alternative

Spearman's rank correlation coefficient (ρ, pronounced "rho") measures the strength and direction of a monotonic relationship between two variables. A monotonic relationship is one where the variables tend to move in the same direction, but not necessarily at a constant rate.

Spearman correlation works by ranking the values of each variable, then calculating the Pearson correlation on the ranks. This makes it:

1. Robust to outliers. Ranks are not affected by extreme values.
2. Capable of detecting non-linear monotonic relationships. If y consistently increases as x increases (even if not linearly), Spearman will detect it.
3. Appropriate for ordinal data. When variables are rankings rather than measurements, Spearman is the natural choice.

import pandas as pd
import numpy as np

# Non-linear monotonic relationship
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y = np.log(x)

df = pd.DataFrame({'x': x, 'y': y})

pearson_r = df['x'].corr(df['y'])
spearman_rho = df['x'].corr(df['y'], method='spearman')

print(f"Pearson r: {pearson_r:.4f}")    # 0.97 (underestimates)
print(f"Spearman rho: {spearman_rho:.4f}")  # 1.00 (perfect monotonic)

The relationship y = log(x) is perfectly monotonic — as x increases, y always increases. Spearman detects this perfectly (ρ = 1.0). Pearson underestimates the relationship (r = 0.97) because the relationship is not linear.

Outliers and Pearson vs Spearman

import pandas as pd
import numpy as np

# Data with an outlier
df_clean = pd.DataFrame({
    'x': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    'y': [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
})

df_outlier = pd.DataFrame({
    'x': [1, 2, 3, 4, 5, 6, 7, 8, 9, 100],
    'y': [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
})

print(f"Clean Pearson: {df_clean['x'].corr(df_clean['y']):.4f}")      # 1.0000
print(f"Outlier Pearson: {df_outlier['x'].corr(df_outlier['y']):.4f}")  # 0.5723

print(f"Clean Spearman: {df_clean['x'].corr(df_clean['y'], method='spearman'):.4f}")      # 1.0000
print(f"Outlier Spearman: {df_outlier['x'].corr(df_outlier['y'], method='spearman'):.4f}")  # 1.0000

A single outlier dropped the Pearson correlation from 1.0 to 0.57. The Spearman correlation remained at 1.0 because the ranks did not change — the outlier is still the largest value, just as it was before.

When to Use Each

Default choice: If you are unsure, compute both. If they agree, the relationship is likely linear. If they disagree, investigate why — there may be outliers, non-linearity, or a non-monotonic relationship.

Statistical Significance of Correlation

A correlation coefficient is a sample statistic. To determine whether it reflects a real relationship in the population (rather than random chance), you need a p-value:

from scipy import stats

x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
y = [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

r, p_value = stats.pearsonr(x, y)
print(f"r = {r:.4f}, p-value = {p_value:.6f}")

rho, p_value_spearman = stats.spearmanr(x, y)
print(f"rho = {rho:.4f}, p-value = {p_value_spearman:.6f}")

A p-value below 0.05 means the correlation is statistically significant — it is unlikely to have occurred by chance. But statistical significance is not practical significance. With a large enough sample, even a correlation of 0.01 can be statistically significant. Always report the correlation coefficient (effect size) alongside the p-value.

The Four Major Pitfalls

Pitfall 1: Correlation does not imply causation.

This is the most important principle in all of statistics. Two variables can be correlated because:

• X causes Y
• Y causes X
• A third variable Z causes both X and Y
• The correlation is a coincidence

import pandas as pd

# Ice cream sales and drowning deaths are correlated
# Does ice cream cause drowning? No — hot weather causes both.
df = pd.DataFrame({
    'ice_cream_sales': [100, 150, 200, 250, 300, 350, 400],
    'drowning_deaths': [2, 3, 5, 7, 8, 10, 12]
})

r = df['ice_cream_sales'].corr(df['drowning_deaths'])
print(f"Correlation: {r:.4f}")  # 0.99 (strong, but not causal)

The correlation is 0.99 — nearly perfect. But banning ice cream would not reduce drowning deaths. The confounding variable is temperature: hot weather increases both ice cream sales and swimming (hence drowning).

Pitfall 2: Outliers create spurious correlations.

A single outlier can create a strong correlation where none exists in the bulk of the data.

import pandas as pd
import numpy as np

# No relationship, but one outlier creates correlation
x = np.random.normal(0, 1, 50)
y = np.random.normal(0, 1, 50)

df_no_outlier = pd.DataFrame({'x': x, 'y': y})
df_with_outlier = pd.DataFrame({
    'x': np.append(x, 10),
    'y': np.append(y, 10)
})

print(f"Without outlier: {df_no_outlier['x'].corr(df_no_outlier['y']):.4f}")  # ~0.0
print(f"With outlier: {df_with_outlier['x'].corr(df_with_outlier['y']):.4f}")    # ~0.8

Pitfall 3: Non-linear relationships are invisible to Pearson.

As shown earlier, a perfect quadratic relationship has a Pearson correlation of zero. Always plot the data.

Pitfall 4: Ecological fallacy — group-level correlation does not apply to individuals.

If countries with higher average chocolate consumption have more Nobel laureates, it does not mean that eating chocolate makes you smarter. Group-level correlations often do not hold at the individual level.

What You Need for Causal Claims

To move from correlation to causation, you need more than a correlation coefficient:

1. Temporal precedence. The cause must precede the effect in time.
2. Control for confounders. Use stratification, matching, or regression to account for other variables that might explain the relationship.
3. A plausible mechanism. There must be a reasonable explanation for how X causes Y.
4. Experimental evidence. Randomized controlled trials are the gold standard. When experiments are not possible, natural experiments or quasi-experimental methods can provide stronger evidence than observational correlation.

Practical Guidelines for Analysts

1. Always plot the data. Scatter plots reveal what correlation coefficients hide.
2. Compute both Pearson and Spearman. If they disagree, investigate why.
3. Check for outliers. Remove them and recalculate to see if the correlation is driven by a few points.
4. Report confidence intervals. A correlation of 0.6 with a 95% CI of [0.55, 0.65] is very different from a correlation of 0.6 with a 95% CI of [0.1, 0.9].
5. Never use the word "causes" based on correlation alone. Use "is associated with" or "is related to" unless you have experimental evidence.
6. Look for confounders. Ask: what third variable could explain this relationship?