Convolution and Pooling

You have an RGB image tensor shaped (3, 32, 32) — CIFAR-10 layout, channels first — and you want to slide a 5x5 filter across it. Before you touch nn.Conv2d, you should be able to answer two questions out loud: how large is the resulting feature map, and how many parameters does the layer hold? Interviewers ask both, and PyTorch will happily disagree with your hand calculation at inference time if you get the first one wrong.

The output size formula is:

H_out = floor((H_in + 2p - k) / s) + 1 — and identically for width, where k is kernel size, p is padding, s is stride.

Work through three cases on a 32x32 input:

Padding of (k-1)/2 with stride 1 reproduces the input size — that is what "same" padding means, and it lets you stack convs without the map shrinking under you. Stride 2 halves each dimension. The floor function matters: when the numerator does not divide evenly, PyTorch drops the final partial position instead of padding it, so your hand-computed shape and the framework's can differ by one when you stack several strided layers.

import torch
import torch.nn as nn

x = torch.randn(1, 3, 32, 32)              # NCHW: batch, channels, H, W
conv = nn.Conv2d(3, 16, kernel_size=5, stride=1, padding=0)
y = conv(x)
print(y.shape)                              # torch.Size([1, 16, 28, 28])

k, p, s = 3, 1, 2
print((32 + 2 * p - k) // s + 1)            # 16 — the formula by hand
print(conv.weight.shape)                    # torch.Size([16, 3, 5, 5])

Now the parameter count. conv.weight holds (out_channels, in_channels, k, k) = 16 x 3 x 5 x 5 = 1,200 weights, plus 16 biases: 1,216 parameters for the entire layer. A fully connected layer over the flattened image would need 3 x 32 x 32 = 3,072 inputs x 16 units = 49,168 weights — 40x more, and every position would need its own copy. That ratio is the whole reason convolutions exist. In Building a Network in PyTorch your inputs were already flat vectors where dense layers make sense; images carry spatial structure, and dense layers throw it away.

Two knobs you will use constantly: padding accepts a tuple (pad_h, pad_w) for asymmetric borders, and dilation stretches the kernel so a 3x3 kernel with dilation 2 reads a 5x5 region without extra parameters. Dilated convolution changes the formula to (H + 2p - d(k-1) - 1)/s + 1, and it is how segmentation networks grow their receptive field without pooling away resolution.

The standard interview follow-up: "what happens when you double the stride?" Each spatial dimension roughly halves, so activations at the next layer drop by about 4x — that trade between resolution and compute follows you through every architecture in CNN Architecture Patterns, so get comfortable computing it on a whiteboard in under a minute.

Two shape errors account for most debugging sessions. First, the channel axis must line up: nn.Conv2d expects (batch, in_channels, H, W), so a grayscale dataset loaded as (N, 1, 28, 28) fed into a layer built with in_channels=3 dies with a weight-shape mismatch, and NHWC-format datasets (TensorFlow exports, most raw image folders) must be permuted to channels-first before touching PyTorch convolutions. Second, in_channels of each conv must equal out_channels of the previous one — the chain of channel counts, not the spatial math, is where a stacked model throws its most confusing error. Trace both axes on paper for your first network; after that, reading model(x) shapes becomes reflexive.

Your layer outputs 16 feature maps. Ask a sharper question: what is each of those 16 filters looking for, and why is the same set of 25 numbers applied at the top-left corner and again at the bottom-right corner of the image?

Parameter sharing is the answer: a convolution uses one weight tensor at every spatial location. When the kernel slides from pixel (0,0) to pixel (27,27), the weights do not change. The gradient for each weight is accumulated across every position where it was used, so a kernel that fires on vertical edges learns from every vertical edge in every image in the batch. Instead of learning 49,168 independent numbers, you learn 1,216 numbers that are reused 784 times each on a 28x28 map. That data efficiency is why CNNs train on thousands of images rather than millions.

Three consequences worth stating precisely, because interviews probe them:

1. Translation behavior. Move a whisker pattern 4 pixels right and the conv response moves 4 pixels right with the same values (borders aside). The filter detects the pattern wherever it appears — no separate weights per location.
2. Inductive bias. You are assuming useful structure is local. True for images: edges, textures, and color blobs are neighborhoods. False for tabular data where column 3 has no spatial relation to column 4 — feeding shuffled columns into a 1D convolution is a standard misuse of the layer.
3. Receptive field growth. The receptive field is the region of the original input that shapes one output value. A first 5x5 conv gives each output a 5x5 view. A second 5x5 conv on that feature map sees 9x9 original pixels (5 + 5 - 1); a third sees 13x13. The field grows linearly with depth for a fixed kernel size, which is the fundamental argument for depth: to recognize a cat, one output must simultaneously see whiskers, ear, and eye, and covering a 224x224 image with 5x5 kernels alone would take dozens of layers. Two standard fixes — small 3x3 kernels stacked deeper, and pooling to jump resolution — are exactly what CNN Architecture Patterns is built around.

You can also inspect what a trained filter responds to, a favorite follow-up: "show me a filter." Pull conv.weight[0] and it typically resembles a Gabor patch — opposing dark and light bands. Simple convolutions trained on natural images converge on edge detectors, color gratings, and blobs regardless of dataset; this was one of the first big visualizations of what deep nets learn.

w = conv.weight.detach()             # (16, 3, 5, 5)
edge = w[0]                          # one filter, all three channels
print(edge.shape)                    # torch.Size([3, 5, 5])
print(edge.mean(dim=(1, 2)))         # per-channel mean near 0 => edge detector
print(edge.abs().sum().item())       # filter "energy" for comparing filters

Push sharing further and you get the depthwise convolution: each input channel gets its own k x k kernel instead of one kernel spanning all channels, and channels are mixed afterwards with 1x1 convolutions. MobileNets cut parameters roughly 8x that way. The mechanism is the same sliding window you just wrote — split into a spatial step and a channel step. The channel-mixing half, the 1x1 convolution, gets its own treatment in the next topic.

One nuance worth having ready: sharing means border positions are not treated exactly like interior positions. With zero padding, a filter touching the image edge multiplies against zeros, so its response is systematically weaker there than the same pattern in the middle — evidence that border pixels are slightly less trusted by the layer, and one reason same padding exists. It is also the honest limitation of the translation claim: responses shift with the pattern only away from the borders.

After a conv layer you often see MaxPool2d(2): a 28x28 feature map becomes 14x14 and the next layer does a quarter of the arithmetic. Pooling is a fixed operation — no weights, nothing learned — and it earns its place for three reasons, while costing you two things you should never forget.

What it buys:

1. Computation. With kernel equal to stride (non-overlapping), a 2x2 pool cuts activations 4x. Compute in the next conv scales with map area, so downsampling before the expensive deep stack is where most of your FLOPs are saved.
2. Translation tolerance. Max over a 2x2 window means a feature shifted by one pixel still produces the same pooled value as long as it stays inside the window. Slight movement no longer changes what downstream layers see.
3. Receptive field per layer. Each pooled step jumps 2 input pixels, doubling coverage without adding parameters — it is the cheapest way to grow the receptive field discussed in the previous section.

The size formula mirrors convolution: with k=2, s=2, p=0, 28x28 becomes 14x14, but a 13x13 map becomes floor((13-2)/2) + 1 = 6, not 7. Remember the floor: odd-sized maps shrink asymmetrically, and errors here compound across a stack.

import torch
import torch.nn as nn

x = torch.randn(1, 64, 28, 28)
pool = nn.MaxPool2d(kernel_size=2, stride=2)             # non-overlapping
print(pool(x).shape)                                     # [1, 64, 14, 14]

same = nn.MaxPool2d(kernel_size=3, stride=1, padding=1)  # keeps size
print(same(x).shape)                                     # [1, 64, 28, 28]

What it costs:

• Spatial precision. After three 2x2 pools, one cell covers an 8x8 input region. Detection and segmentation need that resolution back, which is why those architectures substitute strided or dilated convolutions and skip connections for aggressive pooling.
• Discarded evidence. Max keeps one winner and throws away the runner-ups and their positions inside the window; AvgPool keeps the mean instead — smoother, but weaker for "is this pattern present?" features. Nothing about the choice is stored, and the model cannot recover which of the four cells fired.

That leads to the interview staple: "max pool or stride-2 conv?" The modern answer is that learnable downsampling usually wins — a stride-2 convolution halves resolution while letting the network decide what to keep, and that is what ResNet-class backbones use for most of their stages. Max pooling survives in older stacks (LeNet, VGG), in encoder-decoder architectures that need to route maxima back through unpooling, and in the global variant: nn.AdaptiveAvgPool2d(1) collapses any H x W map to 1x1 per channel, replacing a giant flattening layer — ResNet's classification head turns 7x7x512 = 25,088 values into a clean 512-dim vector.

Two refinements show up in real architectures and interviews. Overlapping pooling — kernel_size=3, stride=2 — leaves a one-pixel overlap between windows; OverFeat reported slightly better accuracy from it, and the size formula still runs with the floor: a 224-wide map gives floor((224-3)/2)+1 = 111 under k=3, s=2 versus floor((224-2)/2)+1 = 112 under k=2, s=2 — compute it rather than remember it. And because pooling has no parameters, it behaves identically in train() and eval() — no running statistics, no stochasticity — so it is never the source of a train/eval mismatch; BatchNorm and dropout are.

One practical trap: pooling halves the map every time you use it, so count your pools against your input size. Four 2x2 pools on a 32x32 input leaves 2x2, and your classifier head collapses with it. With the mechanics down, you are ready to assemble whole networks in CNN Architecture Patterns.