Scalars, Vectors, and Matrices

Debugging an ML pipeline at 2 a.m. usually means staring at a shape error: mat1 and mat2 shapes cannot be multiplied (32x784 and 28x28). The engineers who recover fastest are the ones who can look at any array and immediately say what each axis means. That skill starts with rank: the number of axes an array has, which separates the four structures you will meet constantly.

A scalar is a single number, rank 0. Loss values, learning rates, and accuracy figures are scalars: np.float64(0.531) has ndim == 0. A vector is a 1-D list of numbers, rank 1. One user's feature record (age, purchases, days_since_login) is a vector of length 3, and a model's weight vector has one entry per feature. A matrix is a 2-D grid, rank 2: a batch of user records is a matrix with one row per user and one column per feature. A tensor is the generalization to arbitrary rank: rank 3 for a batch of grayscale images (batch, height, width), rank 4 for color images (batch, channels, height, width), rank 5 for video (batch, channels, time, height, width).

import numpy as np

scalar = np.float32(0.53)            # ndim=0, shape=()
vector = np.array([24.0, 3.0, 7.0])  # ndim=1, shape=(3,)
matrix = np.zeros((4, 3))             # ndim=2, shape=(4 users, 3 features)
tensor = np.zeros((32, 3, 64, 64))    # ndim=4: batch, channels, H, W

for name, a in [('scalar', scalar), ('vector', vector),
                ('matrix', matrix), ('tensor', tensor)]:
    print(name, a.ndim, a.shape)      # rank and shape are all you need

The naming convention matters more than the names. Shape is always read axis by axis, and in ML every axis has a semantic label you should be able to state aloud. In a training matrix of shape (50000, 784), axis 0 indexes training samples and axis 1 indexes pixel features; flattening a 28x28 image into 784 numbers is what let it enter axis 1. In a token tensor of shape (batch, sequence, vocabulary), each axis answers a different question: which item in the batch, which position in the sequence, which token in the vocabulary. When a shape error appears, the fix is almost always a reshape, transpose, or squeeze, and you pick the right one by naming the axes before and after. Adding a new axis with np.newaxis (or None) is the fourth routine move: turning a length-d vector into a (1, d) row lets you subtract it from every row of an (n, d) matrix in one expression, which is exactly how mean-centering works.

One convention trap: libraries label axis 0 differently. NumPy and scikit-learn put samples on axis 0 (batch-major). PyTorch convolutional layers expect channels-first (batch, channels, height, width), while TensorFlow defaults to channels-last. Transposing between them is a routine bug and a routine interview question. The habit that inoculates you: print x.shape at every pipeline boundary and annotate what each axis means in a comment. Rank, shape, and axis meaning are the three facts you need from any array, and once they are habit, shape errors shrink from mysteries into five-second diagnoses. The next chapter develops what it means for a single example to be a point in that space geometrically.

A table of 50,000 users with 20 features is a matrix of shape (50000, 20), but it is also 50,000 points scattered through a 20-dimensional space. That second reading is not decoration; it is the mental model that makes distance-based models, clustering, linear separators, and dimensionality reduction all obvious at once.

Read one row as coordinates. A user with features (age=24, purchases=3, days_since_login=7) is the point (24, 3, 7) in three-dimensional space. Two users are neighbors when their coordinate triples are close, and closeness is measured by Euclidean distance, the straight-line length between the points:

import numpy as np

u = np.array([24.0, 3.0, 7.0])
v = np.array([26.0, 12.0, 1.0])

# Euclidean distance: sqrt of summed squared differences
d = np.sqrt(np.sum((u - v) ** 2))
print(d)                       # 11.0
print(np.linalg.norm(u - v))   # 11.0, the same via the L2 norm

K-nearest neighbors and k-means are nothing but this distance computation repeated across every point in the set. A classifier that votes among the 5 nearest training rows is trusting that geometric proximity in feature space implies label similarity; when it fails in practice, the usual cause is that one feature's scale dominates the distance (income in dollars swamping age in years), which is why standardization accompanies every distance-based model. Rescale features to comparable ranges and the geometry becomes fair.

The point view also makes linear models concrete. A logistic regression computes one number per point: the dot product of the weight vector with the example's coordinates, plus a bias, passed through a sigmoid. Training moves the weight vector until the points of class 1 sit on the positive side of the hyperplane it defines and class 0 on the negative side. Every classification boundary you will study, from a perceptron's flat plane to a neural network's folded surfaces, is still a statement about where points fall relative to directions in this space.

The number of dimensions is itself a modeling decision with a famous pathology. In high dimensions, points become nearly equidistant: with enough independent features, the gap between the nearest and farthest neighbor shrinks to a small fraction of the distances themselves, so the concept of neighbor stops discriminating. This is the curse of dimensionality, and it explains three phenomena you will meet later: why k-NN degrades past a few dozen unscaled features, why clustering in 300 dimensions finds almost nothing without projection, and why PCA is applied before distance computations on wide data. Each new feature adds an axis the data must be spread across, and real datasets rarely fill the new axes with signal; they fill them with noise.

Dimensionality also sets the linear algebra vocabulary for everything ahead. A single example is a length-d vector, the entire training set is an (n, d) matrix, and a linear model computes a dot product between the weight vector and each example vector to produce one number per row. Matrix multiplication is how that happens for all n rows at once, and that efficiency is what makes the matrix representation of a dataset more than a notational convenience.

Deep learning frameworks push the same ideas to higher rank, and their entire performance model depends on how tensors are laid out in memory. Understanding the batch axis and axis conventions turns framework shape errors from mysterious to mechanical.

A tensor generalizes a matrix to any number of dimensions, and frameworks annotate dimensions by role. A typical NLP batch is shaped (batch, sequence, embedding): 32 sentences, each padded to 64 tokens, each token a 300-dimensional embedding vector. A vision batch is (batch, channels, height, width) in PyTorch and (batch, height, width, channels) in TensorFlow, and getting this wrong produces either a clear error or, worse, a model that trains with degraded accuracy because it reads channels as spatial axes. Both conventions describe the same data; they differ only in axis order, and tensor.permute(0, 2, 3, 1) or np.transpose(x, (0, 2, 3, 1)) converts between them explicitly.

import torch

x = torch.randn(32, 64, 300)      # batch, sequence, embedding
print(x.shape, x.ndim)            # torch.Size([32, 64, 300]) 3

# mean-pool over the sequence axis only, keeping the batch axis
pooled = x.mean(dim=1)            # (32, 300)

# per-token normalization over the embedding axis
normed = x / x.norm(dim=2, keepdim=True)   # (32, 64, 1) broadcasts to (32, 64, 300)

# reshape for a linear head: flatten sequence into features
flat = x.reshape(32, -1)          # (32, 19200); -1 infers 64*300

Reductions are where axis labels earn their keep. mean, sum, max, and norm all take an axis (NumPy) or dim (PyTorch) argument naming which axis collapses; keepdim=True preserves the rank so broadcasting still lines up. Attention in transformers is entirely a story of batched matrix products over these axes, and the famous bugs (a softmax run over the batch instead of the sequence) are axis confusions, visible the moment you read shapes aloud. Reshape closes the loop between frameworks and classical models: x.reshape(-1, 300) on a (32, 64, 300) batch flattens each token embedding into a row, handing a (2048, 300) matrix to a linear layer, with the negative dimension inferred so you never compute 64 times 300 by hand. The same negative-inference trick sizes any reshape where one axis is implied by the total element count, which stays constant across every valid reshape of a tensor.

Broadcasting carries over from NumPy and follows the same trailing-axis rule you used for feature matrices: shapes are compared from the right, and any axis of length 1 stretches. A (32, 64, 300) tensor divided by a (32, 64, 1) norm works without a loop, but a (32, 64, 300) tensor against a (64, 300) tensor silently broadcasts over the batch, which is correct for a per-position attention mask and a subtle bug for anything that should have been per-sample. When a shape error or a silent broadcast surprises you, count axes from the right and label each one; the mismatch always names itself. This axis-first reading is the tool you will apply to matrix products in Matrix Multiplication and Transpose, where the same discipline turns derivations into shape bookkeeping.