Matrix Multiplication and Transpose

Every forward pass, every attention layer, and every linear algebra derivation you will meet in an interview reduces to one shape rule. A product of an (m x k) matrix and a (k x n) matrix is defined and yields an (m x n) matrix. The inner dimensions, both labeled k, must match; the outer dimensions, m and n, survive untouched into the result. When they do not match, the operation is undefined, and both NumPy and PyTorch will say so.

What the rule means mechanically: entry (i, j) of the result is the dot product of row i of the left matrix with column j of the right matrix. Each of those is a length-k pairing, which is exactly why both operands must agree on k. The result has m rows (one per left row) and n columns (one per right column).

The rule also explains why the order of operands matters and why multiplying a matrix by its own transpose always works: A of shape (m x k) times A.T of shape (k x m) gives a square (m x m) result, while A times A is undefined unless m equals k. This A.T @ A pattern is not a coincidence; it is how the Gram matrix, the covariance structure of features, and the normal equation all come into being, and you will see it recur across regression, PCA, and attention.

import numpy as np

A = np.random.randn(4, 3)      # m=4, k=3
B = np.random.randn(3, 5)      # k=3, n=5
C = A @ B                      # (4, 5): inner 3s consumed, outer 4 and 5 survive
print(C.shape)                 # (4, 5)
print((A @ A.T).shape)         # (4, 4), square, since inner dims both 3

# inner dimensions disagree: 3 != 4, so this raises
try:
    A @ np.random.randn(4, 2)
except ValueError as e:
    print('shapes (4,3) and (4,2) not aligned:', e)

Three consequences pay rent constantly in ML code. First, multiplication is not commutative: A @ B and B @ A are not even both defined in general, and when both are, the results differ. Second, a matrix can multiply a vector: a (d, d) weight matrix times a length-d vector (treated as a (d, 1) column) returns another length-d vector, which is what one layer does to one example. Third, products of more than two matrices chain left to right only when each adjacent pair shares an inner dimension, so a network's layers must agree on width: a (784, 256) layer feeding a (128, 10) layer fails loudly unless a (256, 128) layer sits between them.

Framework errors are shape-rule messages in disguise. PyTorch's mat1 and mat2 shapes cannot be multiplied (32x784 and 28x28) is telling you the inner dimensions 784 and 28 disagree; the fix is a reshape or a transpose, chosen by which axis should have matched. In an interview, being able to read that message and name the offending axes, without running anything, is a small but reliable signal of fluency. The next chapter shows why the batch-first layout makes one product equal n predictions at once.

Suppose a dense layer maps 784 input features to 256 hidden units and then to 10 output classes. For one example you would compute a length-784 vector times a (784 x 256) weight matrix, add a bias, apply an activation, then repeat. Doing that in a Python loop over 32 examples means 32 iterations of interpreted code and no parallelism. Storing the batch as a (32 x 784) matrix turns all 32 predictions into one product.

The shapes line up precisely because of the inner-dimension rule. X has shape (32 x 784), W1 has shape (784 x 256), so X @ W1 is (32 x 256): one row of hidden activations per example, every row computed simultaneously. Add the bias vector by broadcasting along axis 1, apply an elementwise activation, then multiply by W2 of shape (256 x 10) to get (32 x 10) logits, one row of class scores per example.

import numpy as np

rng = np.random.default_rng(0)
X  = rng.standard_normal((32, 784))     # batch of 32 flattened images
W1 = rng.standard_normal((784, 256)) / np.sqrt(784)
b1 = np.zeros(256)
W2 = rng.standard_normal((256, 10)) / np.sqrt(256)

h = np.maximum(X @ W1 + b1, 0.0)        # (32, 256), ReLU
logits = h @ W2                          # (32, 10): class scores per example

This batch-as-matrix convention is why deep learning is fast: the underlying product calls into BLAS or cuBLAS, which parallelize across the (32 x 256) output grid, so tripling the batch roughly triples throughput rather than tripling latency. It is also why batch size appears in every training conversation: larger batches give the hardware more independent work per launch, at the cost of the gradient-noise trade-off you will study under SGD. The same X @ W computation with a batch of 8 uses the arithmetic units less fully; with a batch of 4096 it saturates them, until memory becomes the binding constraint.

The same convention reappears everywhere. Attention computes Q @ K.T for a batch of sequences: with Q shaped (batch, seq, d) and K shaped (batch, seq, d), transposing K on its last two axes gives (batch, d, seq), and the batched product over the trailing dimensions yields (batch, seq, seq) attention scores, one (seq x seq) matrix per batch item. Linear regression, transformers, and graph neural networks all obey the identical discipline: arrange examples along one axis, features along another, and let a single product do the work of a loop.

The convention also clarifies what a bias vector does and does not do. A bias of length 256 added to a (32 x 256) activation matrix does not participate in the product; it broadcasts across all 32 rows, giving every example the same learned offset per hidden unit. Gradients follow the same shape logic: the (32 x 256) upstream gradient contracts with the (32 x 784) input to produce the (784 x 256) weight gradient, a product you will derive formally once backpropagation arrives. For now the operative habit is already in place: label the axes of every operand, apply the inner-dimension rule, and the rest of deep learning's mechanics becomes bookkeeping.

Transposition swaps rows and columns: the (i, j) entry of A.T is the (j, i) entry of A, so an (m x n) matrix becomes (n x m). Its ML job is almost always the same one: manufacture the matching inner dimension that a product needs. The most famous instance is the normal equation for ordinary least squares, where transposes turn an ill-shaped residual into a solvable system.

Start from the design matrix X of shape (n x d) and targets y of shape (n,). The least-squares objective is the squared norm of residuals X w - y. The critical observation: X w requires w to have length d, and (X w - y) is length n, so its square is fine, but differentiating with respect to w must produce a length-d gradient. The transposes in the derivation enforce exactly that bookkeeping:

J(w) = (X w - y)^T (X w - y)          # (1 x n)(n x 1) = scalar
grad: dJ/dw = 2 X^T (X w - y)         # (d x n)(n x 1) = length d
set to zero:  X^T X w = X^T y         # (d x d)(d) = (d x 1)
w = (X^T X)^{-1} X^T y               # shapes: (d x d)(d x d)(d x 1) = (d,)

Read the shapes at every line and the derivation becomes mechanical. X.T is (d x n), so X.T @ X is (d x d), a square matrix that can be inverted when the columns of X are not perfectly collinear. If two features are exact copies, X.T @ X is singular, the inverse does not exist, and that singularity is the same collinearity that breaks multiple-regression coefficient interpretation; ridge regression fixes it by adding a multiple of the identity, which changes no shapes and guarantees invertibility.

Backpropagation is the same shape discipline applied layer by layer. If a layer computes h = X @ W with W of shape (d x k), then the gradient of the loss with respect to W must have shape (d x k), and it arrives as X.T @ dL/dh: (d x n)(n x k) = (d x k). The transpose is not decorative; it is the only arrangement of the two factors whose inner dimensions line up. This is why derivations that skip transposes lose readers: every symbol carries a hidden shape, and the shapes are the proof that the algebra is consistent.

Two properties round out the toolbox and surface as interview questions. Transposition distributes over addition and reverses product order, (A B).T = B.T A.T, which is why gradients through composed layers come out transposed relative to the forward pass. And matrix multiplication is associative but not commutative: (A B) C = A (B C) whenever the products exist, but A B generally differs from B A and often is not even defined. Knowing that a transpose was inserted to fix an inner dimension, rather than to look elegant, is the difference between reciting the normal equation and understanding it. Next you will use these products to measure similarity: Dot Products, Norms, and Projections.