Dot Products, Norms, and Projections

Two vectors, one number. The dot product of u and v (both length d) is the sum of their term-by-term products, and it is simultaneously an algebraic sum and a geometric statement about direction. Holding both readings at once is what makes similarity, projections, and attention click.

Algebraically: u . v = u1v1 + u2v2 + ... + ud*vd. Geometrically: u . v = ||u|| ||v|| cos(theta), where theta is the angle between the vectors and ||.|| is the length. The two formulas compute the same number, and the second one explains everything interesting about the first. When the vectors point in the same direction, cos(theta) = 1 and the product equals the product of their lengths. When they are perpendicular, or orthogonal, cos(theta) = 0 and the dot product is exactly zero, regardless of either vector's length. Opposite directions give a negative product.

import numpy as np

u = np.array([1.0, 0.0])
v = np.array([0.0, 3.0])
w = np.array([2.0, 2.0])

print(u @ v)                                  # 0.0, orthogonal
print(u @ w, v @ w)                           # 2.0 6.0, both positive: acute angles

# recover the angle from the definition
cos_t = (u @ w) / (np.linalg.norm(u) * np.linalg.norm(w))
print(np.degrees(np.arccos(cos_t)))           # 45.0 degrees

ML runs on this identity constantly. A linear model's output for one example is the dot product of the weight vector with the feature vector: features aligned with the weights contribute positively, orthogonal features contribute nothing, and opposing features subtract. Training is the search for a weight vector whose alignment pattern separates the classes. In embedding spaces, dot products between learned vectors are the similarity scores behind recommendation and retrieval: item embeddings trained so that a user vector and a relevant item vector point in nearly the same direction, and a maximum-dot-product lookup over the index returns the ranking.

The magnitude half of the identity matters as much as the angle. A vector ten times longer produces a ten-times-larger dot product at the same angle, which means raw dot-product scores silently mix similarity with popularity or verbosity. Ranking systems that ignore this bias toward long or frequent items, and the correction is always the same: normalize, or model the magnitude explicitly as its own feature.

Orthogonality itself is a design tool. One-hot encodings are pairwise orthogonal, so their dot products are zero for different categories and one for the same category, which is precisely why they inject no spurious similarity between unrelated categories. The centered data matrix in PCA is built so its principal component columns are orthogonal, and attention heads learn query and key subspaces whose dot products measure relevance between positions. Zero dot product means no overlap in what two directions capture, and in interview settings being able to state that identity, ||u|| ||v|| cos(theta), and use it to explain a similarity score is worth more than any formula-sheet recital. The next chapter examines the other half of that identity: the lengths themselves, measured by norms.

Projections are where the identity turns into an operation. The scalar projection of u onto v is the length of u's shadow on v's direction: u . v / ||v||. The vector projection — the shadow itself — scales that by v's direction:

import numpy as np

u = np.array([3.0, 4.0])
v = np.array([1.0, 0.0])

scale  = (u @ v) / (v @ v)      # 3.0  — how far along v u reaches
proj   = scale * v              # [3., 0.] — the component of u ALONG v
rej    = u - proj               # [0., 4.] — the leftover, orthogonal to v

assert np.isclose(rej @ v, 0.0)          # rejection is perpendicular to v
assert np.isclose(np.linalg.norm(u) ** 2,
                  np.linalg.norm(proj) ** 2 + np.linalg.norm(rej) ** 2)  # Pythagoras

This decomposition is the mechanical core of a great deal of ML. PCA finds the directions that capture the most variance and projects data onto them; least-squares regression is the projection of the target vector onto the column space of the design matrix, with the residual exactly orthogonal to it; Gram-Schmidt orthogonalization builds a basis by repeatedly subtracting projections; attention computes, per query, how much of each key direction to take. In every case the same two-line operation does the work: measure alignment with a dot product, scale by the target's length, subtract to get the perpendicular remainder. The Pythagorean identity in the assert is why the split is lossless — nothing about u is discarded, only separated into "along v" and "orthogonal to v," which is the language that the norms in the next chapter and the eigenvectors in the chapter after them both speak.

A norm measures a vector's length, and the family you will meet is the Lp norms. The L2 norm, ||u||2 = sqrt(u1^2 + ... + ud^2), is the familiar straight-line length from the origin, the Euclidean distance you have already used for clustering. The L1 norm, ||u||1 = |u1| + ... + |ud|, sums absolute values instead of squares. They agree on axis-aligned vectors and diverge everywhere else, and that divergence is why they power different ML mechanisms.

The geometric difference shows up in the set of vectors at a fixed norm. All vectors at L2 distance 1 from the origin form a round sphere; all vectors at L1 distance 1 form a diamond whose corners sit on the axes. Optimization constrained to L1 therefore touches solutions on those corners, where one or more coordinates are exactly zero. That is the entire reason Lasso (L1 regularization) produces sparse weight vectors: the constraint geometry forces coordinates to zero, performing feature selection as a side effect of the geometry rather than as a separate filtering step. L2 (ridge) pulls weights toward zero but almost never lands on exactly zero, shrinking correlated features together instead of choosing between them.

import numpy as np

u = np.array([3.0, -4.0])
print(np.linalg.norm(u))                 # 5.0, the L2 norm
print(np.abs(u).sum())                   # 7.0, the L1 norm
print(np.abs(u).max())                   # 4.0, the L-infinity norm: largest component

# normalizing to unit length turns any nonzero vector into a direction
unit = u / np.linalg.norm(u)
print(np.linalg.norm(unit))              # 1.0

Unit normalization deserves special attention because of what it does to dot products. Divide any vector by its L2 norm and you get a unit-length vector along the same direction; the dot product of two unit vectors is exactly cos(theta), bounded between -1 and 1. This is the bridge between dot products and cosine similarity, and it is why embedding systems normalize vectors before scoring: after normalization, raw dot product and cosine similarity become the same number, and magnitude can no longer dominate a ranking.

Norms also bound the quantities that training controls. Gradient clipping is an L2-norm operation (rescale the gradient whenever its norm exceeds a threshold), weight decay adds an L2 penalty to the loss, and numerical stability arguments throughout linear algebra are written in terms of norms because norms convert matrix manipulations into scalar inequalities you can reason about. Batch normalization and layer normalization are, name and all, norm computations: they rescale activation vectors so downstream layers see inputs of controlled length, which keeps gradients in a workable range across depth.

The distance formulas built on norms are equally load-bearing. Euclidean distance is the L2 norm of the difference vector; Manhattan distance is the L1 norm of the difference and behaves better on high-dimensional data where the L2 metric loses contrast; the L-infinity norm reports the single largest coordinate difference, useful for worst-case tolerance checks. Choosing the norm is choosing what counts as far apart, and in interviews you should be ready to justify that choice the way you would justify a loss function.

The general Lp norm is ||u||_p = (sum |u_i|^p)^(1/p), and the family has two edges worth knowing. The L-infinity norm (p -> inf) is the max-absolute-coordinate, shown above — it answers "what is the single worst deviation?" rather than "how far overall?". The p = 1 corner is a true norm, but the p = 0 case is not: counting non-zero entries violates the triangle inequality, so ||.||_0 is a cardinality measure that Lasso approximates rather than optimizes. Stating that distinction cleanly separates a memorized fact from a understood one.

import numpy as np

z = np.array([0.0, 0.0])
# u / ||u||  divides by zero — normalizing a zero vector is undefined, not 1.0
print(np.linalg.norm(z))                        # 0.0

big = np.array([1e8, 1e8])
print(np.linalg.norm(big))                      # 1.414e8 — fine here...
print(np.linalg.norm(big.astype(np.float32)))   # overflow risk: squares exceed 3.4e38

That second example is the numerical trap: the naive L2 formula squares each coordinate, so squaring can overflow float32 long before the norm itself would be large. Production code and LAPACK use a scaled algorithm (find the largest absolute value, factor it out, rescale) — scipy.linalg.norm and np.linalg.norm on float64 sidestep it for realistic inputs, but computing np.sqrt(np.sum(x * x)) by hand in float32 does not. Similarity pipelines hit the same class of bug in reverse: a zero vector makes cosine similarity a 0/0, so real systems define it as 0 (or reject the document) instead of returning NaN — one more reason the guard belongs at the boundary where vectors are produced, not at every call site that scores them.

A search index holds one TF-IDF vector per document in a vocabulary of 100,000 terms. Two documents about the same topic may share only a few dozen non-zero entries, and their lengths vary wildly: a 2,000-word review and a 200-word comment on the same product. Comparing them with Euclidean distance measures topic mismatch and document length at the same time, and length wins. Cosine similarity compares direction only, which is what topic similarity actually means here.

Cosine similarity is the dot product divided by the product of the two norms: cos(u, v) = (u . v) / (||u|| ||v||). It ignores magnitudes entirely; a vector and its 10x multiple have cosine similarity 1.0. The failure mode it fixes is concrete: with raw Euclidean distance, the short comment sits closer to every other short document regardless of topic, because length differences swamp term-overlap differences. Cosine fixes that by projecting each vector onto the unit sphere and measuring the angle there. The two metrics are related by a formula worth knowing: squared Euclidean distance equals 2 - 2*cos(u, v) once both vectors are unit length, so on the sphere the rankings agree; the disagreement only ever comes from magnitude.

In a real retrieval pipeline the choice appears in three places. First, the vectorizer: TF-IDF weighting already suppresses common terms so the dot product emphasizes shared distinctive words. Second, the normalization step, often applied at index time so every stored vector is unit length. Third, the query path, where the query is embedded, normalized, and scored against the index by the same inner-product kernel. Approximate nearest-neighbor libraries then accelerate the scan with quantization and graph indexes, but the similarity definition underneath remains the cosine score you chose here.

import numpy as np

docs = {
    'long_review': {'great': 2.0, 'battery': 1.4, 'screen': 0.9},
    'short_comment': {'great': 0.7, 'battery': 0.5},
    'unrelated': {'policy': 1.8, 'shipping': 1.5},
}
vocab = ['great', 'battery', 'screen', 'policy', 'shipping']
X = {k: np.array([v.get(t, 0.0) for t in vocab]) for k, v in docs.items()}

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(cosine(X['long_review'], X['short_comment']))   # 0.99, same topic
print(cosine(X['long_review'], X['unrelated']))       # 0.0, orthogonal terms

eu = lambda a, b: np.linalg.norm(a - b)
print(eu(X['long_review'], X['short_comment']),
      eu(X['long_review'], X['unrelated']))            # length can invert the ranking

The choice between the two metrics is a modeling decision with a clean rule of thumb. Use cosine when vector magnitude carries no meaning you want to model (topic similarity in text, semantic similarity of embeddings, user-item alignment in recommendation) and when vectors are sparse and length varies. Use Euclidean distance when magnitude is signal: customer records where spend scale separates segments, sensor vectors where amplitude matters, or embeddings explicitly trained so that distance encodes something. A common production pattern is to L2-normalize embeddings at the end of the encoder and then use either metric, since for unit vectors the dot product already equals cosine similarity and nearest-neighbor search reduces to a maximum inner product lookup, which is exactly the operation the next chapter's decompositions exploit at scale.

The failure mode deserves a number, because "length dominates" is easy to assert and easy to forget. Take two documents that share every topic term, one five times longer:

import numpy as np

def tf(term_counts):                      # raw counts, no normalization
    return np.array([term_counts.get(t, 0.0) for t in ['ml', 'rust', 'policy']])

long_  = tf({'ml': 10, 'rust': 5, 'policy': 1})     # 16 terms, mostly on topic
short_ = tf({'ml': 1, 'rust': 1, 'policy': 1})      # 3 terms, half on topic
noise  = tf({'ml': 0, 'rust': 0, 'policy': 9})      # off-topic, long

dot = lambda a, b: a @ b
cos = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(dot(long_, noise), dot(long_, short_))   # 9.0 16.0 — noise still scores 9
print(cos(long_, noise), cos(long_, short_))   # 0.09 0.82 — topic wins

Raw dot product ranks the off-topic document at 56% of the on-topic one purely because it is long; cosine drops it to 0.09 while the genuinely similar short document scores 0.82. That is the whole argument in four printed numbers, and it is the reason an evaluation harness should compare the two metrics on your own data rather than assume either.

The score ranges are worth keeping straight for interviews: cosine lies in [-1, 1] where 0 means unrelated and 1 means identical direction (negative means opposed — rare in non-negative TF-IDF, possible in centered or learned embeddings); a raw dot product is unbounded and, for non-negative inputs, always ≥ 0. And for unit vectors the distinction disappears entirely: dot equals cosine, so normalizing at index time converts an inner-product system into an angular one without touching a single scoring call site. When you are asked when Euclidean and cosine give different rankings, the complete answer is: whenever the vectors differ in magnitude — on the unit sphere the two orderings provably coincide, because ||a - b||² = 2 - 2·cos(a, b).