Eigenvalues and SVD

Multiply a matrix by most vectors and the result points somewhere new: rotated, stretched, sheared. Multiply by an eigenvector and something rare happens. The output points in exactly the same direction as the input, differing only by a scalar factor. Formally, for a square matrix A, a nonzero vector v with Av = lambda v is an eigenvector and lambda is its eigenvalue. The transformation leaves the line spanned by v invariant; it only rescales along it, by the factor lambda.

That equation is the whole definition, and both halves carry meaning. The direction v is the axis the transformation genuinely acts on; the eigenvalue lambda says how much it stretches (lambda greater than 1), compresses (between 0 and 1), flips and stretches (negative lambda), or annihilates (lambda equals zero, meaning v lies in the null space). A matrix with many near-zero eigenvalues is collapsing most of the space onto a lower-dimensional subspace, and that observation is the seed of every compression and denoising technique built on spectral methods. Note also that scaling the eigenvector by any constant yields another eigenvector with the same eigenvalue, so implementations return unit-length versions and leave the overall scale to you.

import numpy as np

A = np.array([[2.0, 1.0],
              [1.0, 2.0]])
vals, vecs = np.linalg.eig(A)
print(vals)                                  # [3. 1.]

v = vecs[:, 0]                               # first eigenvector
print(np.allclose(A @ v, vals[0] * v))       # True: Av = 3v, same direction
print(np.linalg.norm(v))                     # 1.0, implementations return unit vectors

Where do eigenvectors appear in ML? In covariance and correlation matrices, which are symmetric and therefore guaranteed to have real eigenvalues and orthogonal eigenvectors. In the Hessian matrix of second derivatives, whose eigenvalues tell you the curvature of the loss surface in each direction and whether a critical point is a minimum, maximum, or saddle; a Hessian with at least one negative eigenvalue is a saddle regardless of the loss value there. In spectral clustering, where the graph Laplacian's eigenvectors embed nodes so that communities separate. In the PageRank-style centrality computations that seed recommendation baselines. The unifying idea: whenever a symmetric operator describes how data varies or how a loss curves, its eigenvectors pick out the natural axes of that system and its eigenvalues rank them by importance, which is why one definition powers dimensionality reduction, curvature analysis, and clustering alike.

One practical property matters for numerical work: eigenvectors are defined only up to sign, and their order is not guaranteed across libraries or versions. PCA implementations standardize both, and any pipeline that stores eigenvectors as features must handle the sign flip explicitly or accept that retraining can flip a component's orientation between runs, which breaks any model that reads a component's raw value rather than its magnitude. A second property rounds out the picture: symmetric matrices have orthogonal eigenvectors, so distinct eigenvectors of the covariance matrix are guaranteed to be perpendicular, which is exactly the decorrelation property PCA relies on. With the geometric picture set, the next chapter assembles the pieces into the single most-used spectral method in classical ML.

Principal component analysis answers one question with linear algebra: in which directions does the data vary most? The mechanics follow directly from the eigenvector definition. Center the data so every feature has mean zero, compute the covariance matrix C = (1/(n-1)) X.T X, take its eigenvectors, and sort by eigenvalue descending. The first eigenvector is the direction of maximum variance; each subsequent one is the direction of maximum remaining variance subject to being orthogonal to the ones before it.

Why variance equals information is worth stating plainly. A feature that takes the same value for every customer tells a model nothing; a feature that varies does. PCA does not know labels, so for it, variance is the only proxy for signal available, and the eigenvalues quantify each component's share: eigenvalue lambda_i is the variance along component i, and the ratio lambda_i / sum(lambda) is the fraction of total variance that component preserves. The eigenvectors of the symmetric covariance matrix come out orthogonal, so the new components are pairwise uncorrelated coordinates, which is what lets you drop the least informative ones without entangling what remains.

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X = load_iris().data            # (150, 4)
Xs = StandardScaler().fit_transform(X)   # PCA on features of unequal scale lies
pca = PCA(n_components=2).fit(Xs)

print(pca.explained_variance_ratio_)     # e.g. [0.73 0.23]: 96% in two components
Z = pca.transform(Xs)                    # (150, 2) new coordinates
print(pca.components_[0])                # first principal direction, length 4

Standardizing before PCA is not optional when features use different units; otherwise the largest-scale feature dominates the covariance matrix and the first component simply recovers it. After projection, the (n x k) score matrix Z feeds downstream models: visualization in two or three dimensions, denoising by keeping high-variance components and discarding the rest as noise, or dimensionality reduction before a distance-based method, which is where the curse of dimensionality you met earlier gets tamed. The components themselves are also inspectable: each one is a length-d weight vector over the original features, so printing the largest absolute entries of component 1 tells you what that direction physically represents, whether it is a price-versus-quality contrast in survey data or a first-versus-second-order motion contrast in sensor streams.

Choosing k uses the explained-variance curve. Plot the cumulative ratio against component count and find the elbow: the point where each extra component buys little retained variance. A common target is 90-95% cumulative explained variance, but the number is a business decision: image reconstruction might need 95%, while a visualization needs only 2, and a preprocessing step before clustering might sit anywhere between. Two cautions belong in any honest description. First, components are linear; nonlinear manifolds (a Swiss roll, curved image classes) need t-SNE or UMAP instead. Second, PCA is unsupervised, so a component with high variance may be pure nuisance (illumination in faces, session length in clickstreams) and the label-relevant direction may hide in a low-variance component, which is why supervised alternatives exist. The eigenvector route works because the covariance matrix is symmetric; the next chapter shows the decomposition that works for every matrix, rectangular or not.

Eigendecomposition needs a square matrix. The singular value decomposition handles any n x d matrix and is the engine actually running underneath PCA, latent semantic analysis, and matrix completion. Any real matrix X factors as X = U Sigma V^T, where U is (n x n) orthogonal (its columns are left singular vectors), Sigma is (n x d) diagonal with non-negative singular values sorted descending, and V^T is (d x d) orthogonal (its rows are right singular vectors).

The three factors each describe a piece of X's structure. The right singular vectors V are the input directions: the first one is the direction in feature space along which the data varies most, which is why the columns of V are exactly the principal components PCA returns. The singular values sigma_i measure the strength of each direction: variance along component i is sigma_i^2 / (n-1). The left singular vectors U give the coordinates of the samples in that rotated space. Together they say: X acts like stretching along V by Sigma and relabeling the rows via U. The count of nonzero singular values is the rank of X, so SVD reads the effective dimensionality of the data directly off the spectrum, and how fast the values decay tells you whether truncation will be cheap or lossy.

import numpy as np
from sklearn.decomposition import TruncatedSVD

X = np.random.default_rng(0).standard_normal((1000, 50))
U, s, Vt = np.linalg.svd(X, full_matrices=False)
print(U.shape, s.shape, Vt.shape)        # (1000, 50) (50,) (50, 50)

# rank-k truncation: keep the k strongest directions
k = 5
X_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]
print(np.linalg.norm(X - X_k))           # best rank-k error (Eckart-Young)

# sparse-friendly version used in production NLP
svd = TruncatedSVD(n_components=50).fit(tfidf_matrix)

Truncation is where SVD earns its reputation. The Eckart-Young theorem states that keeping the top k singular values and discarding the rest gives the closest rank-k matrix to X in both the Frobenius and spectral norms. That single fact explains three applications. Image compression stores a few hundred singular values and vectors instead of every pixel, reconstructing an approximation whose error equals the discarded singular values. Latent semantic analysis truncates the term-document matrix so synonymy and topic structure emerge in the reduced space, which is how early search systems grouped related queries. Recommendation truncates the user-item matrix for matrix factorization, and the missing entries get filled by the low-rank reconstruction, which is precisely the collaborative-filtering setup.

SVD also delivers the pseudoinverse for free, which handles rectangular and rank-deficient systems where the normal equation's inverse does not exist: minimum-norm least squares solutions come from cutting tiny singular values, and ridge regression can be read as doing exactly that with a floor set by the regularization strength. Numerically, SVD is the trusted route because it is stable where explicit eigendecomposition of X.T X squares the condition number and loses precision. That conditioning argument is the practical reason libraries default to SVD-based solvers for least squares. For interviews, the three-beat answer is: eigenvectors find the axes, PCA picks them by variance on the covariance matrix, SVD computes the same axes for any matrix while revealing rank and strength through the singular values.