Distance and Similarity Metrics

Two on-call engineers are comparing sensor readings, a recommender is comparing user vectors, and a search system is comparing embeddings — all three need the same primitive: a number that says how far apart two vectors are. The mistake beginners make is treating that number as universal. It is a design choice, and the choice decides which neighbors your model sees.

All three common metrics live in one family. For vectors x and y in d dimensions, the Minkowski distance of order p is:

D_p(x, y) = ( Σᵢ |xᵢ − yᵢ|^p )^(1/p)

Work it once by hand so the differences are concrete. Take x = (1, 2) and y = (4, 6):

Euclidean:  √(3² + 4²)     = √25  = 5.0
Manhattan:  |3| + |4|             = 7.0
Minkowski p=3: (27 + 64)^(1/3)    ≈ 4.50
Chebyshev:  max(3, 4)             = 4.0

Same two points, four different answers — and whichever one you pick, every downstream "nearest neighbor" changes with it. Euclidean equals Manhattan here only by coincidence of the 3-4-5 triangle; in general they disagree, and they disagree most exactly where your data is sparsest.

import numpy as np
from scipy.spatial.distance import minkowski, cityblock, euclidean

x, y = np.array([1.0, 2.0]), np.array([4.0, 6.0])
print(euclidean(x, y))        # 5.0
print(cityblock(x, y))        # 7.0  (Manhattan)
print(minkowski(x, y, p=1))   # 7.0  — p=1 IS Manhattan
print(minkowski(x, y, p=2))   # 5.0  — p=2 IS Euclidean
print(np.max(np.abs(x - y)))  # 4.0  — Chebyshev, p=inf

Knowing when each fails is the interview-grade part. Euclidean punishes outliers quadratically — one coordinate off by 100 contributes 10,000 to the sum, so with noisy heavy-tailed features a single extreme reading can dominate the ranking. Manhattan grows only linearly with the same deviation, which is why it is often the more robust default on sparse, noisy, high-dimensional data. Euclidean also assumes the axes are commensurable: add income_in_dollars next to age_in_years and the income term drowns every other coordinate — a scaling failure, not a formula failure, and one that standardization fixes before any metric is computed. One connection interviewers like: Euclidean distance is the L2 norm of the difference vector, and least-squares regression is L2 on the residuals — the same geometry, wearing a different hat.

from sklearn.metrics.pairwise import pairwise_distances
D = pairwise_distances(X, metric="minkowski", p=1)   # the whole n×n matrix at once

The metric argument you pass here recurs across the ecosystem — kNN, DBSCAN, hierarchical clustering all accept the same names — so this single choice propagates to every distance-based component in your pipeline.

Chebyshev has a genuine use case — when a constraint really is bottleneck-shaped ("the slowest machine decides the job time") — but as a similarity measure it is destructive: all 100 dimensions collapse into the one coordinate that happened to be extreme.

One subtlety worth having ready: all three L_p distances are dominated by dimensionality, not just by scaling. In high dimensions, independent coordinates accumulate and pairwise distances spread ever more narrowly relative to their mean — the concentration effect that quietly kills distance-based models, and that we will quantify in the third chapter.

For embedding-space comparison, none of these may be the right question at all: when two vectors point the same way but differ wildly in length, you do not want distance, you want orientation. That is cosine similarity's job.

Two articles cover the identical topic; one is 400 words and the other 4,000. Under Euclidean distance they are miles apart — every coordinate of the longer term-frequency vector is bigger. Under cosine similarity they are nearly identical, because cosine throws away length and compares only the direction of the vectors:

cosine_similarity(x, y) = (x · y) / (‖x‖ · ‖y‖)      ∈ [−1, 1] for signed vectors
cosine_distance(x, y)    = 1 − cosine_similarity(x, y)

Compute it once by hand. x = (1, 2, 3), y = (4, 5, 6):

x·y       = 1·4 + 2·5 + 3·6        = 32
‖x‖       = √(1 + 4 + 9)           ≈ 3.742
‖y‖       = √(16 + 25 + 36)        ≈ 8.775
cos sim    = 32 / (3.742 × 8.775)  ≈ 0.975   → cosine distance ≈ 0.025
Euclidean  = √(3² + 3² + 3²)       ≈ 5.196

The vectors agree strongly in direction while sitting 5.2 units apart in space — exactly the situation TF-IDF document vectors and normalized embeddings are constantly in. Scaling y by any positive constant (the whole document made 10× longer) leaves the cosine similarity untouched:

from sklearn.metrics.pairwise import cosine_similarity, cosine_distances
import numpy as np

docs = np.array([[1, 2, 3], [4, 5, 6], [-1, -2, -3]])
print(cosine_similarity(docs))          # [[1.0, 0.975, -1.0], ...]
print(cosine_distances(docs))           # 1 − sim: [[0.0, 0.025, 2.0], ...]
print(cosine_similarity(docs[:1], docs[:1] * 10))  # still [[1.0]] — length-invariant

The inverse vector (−1, −2, −3) scores −1: same line, opposite direction. For non-negative inputs such as raw counts or TF-IDF, similarity is confined to [0, 1], so cosine distances bunch in a narrow band — great for ranking top-k, less useful for thresholding "these are not similar at all."

Two failure modes to keep straight, because interviewers probe them. First, cosine is the wrong tool when absolute magnitude carries meaning: monthly spend vectors (50, 100) and (500, 1000) have similarity 1.0, yet one customer is worth ten times the other — for churn or LTV problems that identity destroys your signal. Second, the zero vector has undefined cosine (0/0): a document matching no terms in your vocabulary, or an all-zero interaction row, needs an explicit policy instead of a NaN silently entering a ranking.

One geometric footnote for precision: 1 − cosine does not satisfy the triangle inequality, so it is a dissimilarity rather than a true metric — some algorithms (certain KD-tree variants, metric-space indexes) require a real metric. The chord distance √(2(1 − cos)) — the straight-line distance between the vectors after L2-normalizing them — is one, and it is monotone in cosine, so rankings are identical. That equivalence is also practical: for unit-normalized vectors, ‖x − y‖² = 2(1 − cos(x, y)) — normalize first and cosine and Euclidean ranking are interchangeable, which rescues pipelines whose index only speaks Euclidean.

Whatever metric you choose, one invisible assumption has already been made: that each feature's units deserve equal vote in the formula. Data almost never cooperates with that assumption.

Here is a two-row dataset for predicting a purchase: A = (50000, 30), B = (52000, 32) — income in dollars and age in years. Euclidean distance between A and B:

√((52000 − 50000)² + (32 − 30)²) = √(4,000,000 + 4) ≈ 2000.0

Age contributes 4 units to 4,000,004. The feature with the biggest units wins every distance comparison, not because it is more informative but because it is denominated differently. Any model built on raw distances here is an income model wearing a two-feature costume — kNN will find "nearest neighbors" purely by income proximity, and clustering will slice the data into income bands.

The fix is fitting a scaler on the training split only, then applying it everywhere:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

pipe = Pipeline([
    ("scale", StandardScaler()),   # z = (x − μ) / σ, μ and σ from TRAIN rows only
    ("knn", KNeighborsClassifier(n_neighbors=7, metric="minkowski", p=2)),
])
pipe.fit(X_train, y_train)         # scaler transforms train and future queries alike
print(pipe.score(X_test, y_test))

StandardScaler gives every feature mean 0 and variance 1, so one standard deviation of income now weighs the same as one standard deviation of age. Alternatives each encode a different assumption: MinMaxScaler when you need bounded [0, 1] inputs and have no heavy outliers, RobustScaler (median/IQR) when outliers would distort μ and σ themselves, and L2-normalization when only direction should matter (cosine's usual companion). The one rule with no exceptions: fit the scaler on training data only — computing μ and σ over the test set is a small, silent leakage that inflates every downstream metric.

The failure is watchable, not theoretical: on the raw data above, k = 7 returns seven "neighbors" whose incomes sit within a few hundred dollars of the query while their ages differ by 15 years — age contributes nothing, and your two-feature model is secretly a one-feature model. After standardization the neighborhood mixes both axes in proportion to their true variance, and the decision boundary visibly moves. The companion trap is order of operations: fitting the scaler on the full dataset before train_test_split leaks test-set statistics into training, usually by a fraction of accuracy point, usually unnoticed.

Scaling is not universally required, and knowing when to skip it is part of the answer. Trees split on thresholds, so they are invariant to any monotone rescaling — feature importance work like Tree-Based Feature Importance never needed a scaler. Distance models (kNN, SVM, k-means, any L_p metric) and gradient-based optimizers do need it. When an interviewer asks "would you standardize here?", the real question is whether your model's decision rule is expressed in units or in orderings.

Scaling fixes units, but it cannot fix dimensionality. As d grows, something worse than domination appears: in a 100-dimensional space of standardized features, the difference between the nearest and farthest point to a query shrinks toward zero — distances concentrate, so "nearest" loses its meaning. A standard illustration: sample random points uniformly in d dimensions and compute max_dist / min_dist; it lands near 1.0 by d ≈ 100, and at that point every neighbor is essentially tied. Meanwhile the volume of the space grows exponentially, so your n points become exponentially sparse — the curse of dimensionality. Practical consequences: drop useless columns before trusting distances (each noise feature adds a noise term to every metric), prefer cosine on high-dimensional sparse vectors (it measures orientation, which concentrates less pathologically than magnitude), and remember that kNN in particular inherits all of this at prediction time — which is exactly where k-Nearest Neighbors Classification begins.