CNN Architecture Patterns

Every CNN you will meet in interviews or production is a variation on one repeating recipe: convolution -> nonlinearity -> downsampling, then a classifier head at the end. LeNet-5 (1989) already had it: 32x32 input, a 5x5 conv producing 6 maps at 28x28, a pool to 14x14, another conv to 10 maps at 10x10, a pool to 5x5, then fully connected layers to 10 digits. Modern CIFAR networks are the same skeleton with 3x3 kernels and BatchNorm:

block = nn.Sequential(
    nn.Conv2d(3, 32, kernel_size=3, padding=1),   # 32x32 -> 32x32
    nn.BatchNorm2d(32),
    nn.ReLU(inplace=True),
    nn.MaxPool2d(2),                              # 32x32 -> 16x16
    nn.Conv2d(32, 64, kernel_size=3, padding=1),  # 16x16 -> 16x16
    nn.BatchNorm2d(64),
    nn.ReLU(inplace=True),
    nn.MaxPool2d(2),                              # 16x16 -> 8x8
)

x = torch.randn(1, 3, 32, 32)
print(block(x).shape)                             # torch.Size([1, 64, 8, 8])

Two design rules explain almost every choice in that block.

The nonlinearity is not optional. Stack five convolutions with no activation and the composition is still a single linear map — five linear functions composed collapse to one, so depth would buy zero extra expressive power. ReLU between convs is what lets each layer carve a separate region of input space. This is the most common "why" question on the topic: why not just stack convs?

Double the channels whenever you halve the spatial size. Conv compute scales with H * W * C_in * C_out * k * k. Pooling quarters H * W; doubling both channel counts multiplies C_in * C_out by 4. The two cancel, so each block does roughly the same FLOPs as the one before while trading resolution for feature richness. That is why the shape trace looks like a staircase:

padding=1 with k=3 ("same") keeps resolution stable until you deliberately pool — the same shape arithmetic you worked through in Convolution and Pooling — and without it every conv would shrink the map and you would run out of pixels after four layers.

Sizing the stack follows the input. A 32x32 CIFAR input survives three pools (32 -> 16 -> 8 -> 4) with channel widths 32/64/128; a 224x224 ImageNet input can afford five (224 -> 112 -> 56 -> 28 -> 14 -> 7) with 64/128/256/512/512. The rule of thumb: stop pooling when the map hits about 4x4 to 7x7 — push further and you are computing convolutions on a 1x1 grid. BatchNorm carries one operational warning that bites in practice: its running statistics are unreliable with tiny batches, so if your batch size is 4 because the images are large, freeze the statistics or switch to GroupNorm — otherwise training loss looks healthy while eval-mode accuracy quietly collapses. BatchNorm shows up after conv because it stabilizes the activation scale layer by layer, letting you use higher learning rates; it also means you must remember model.eval() at inference, or your batch statistics go sideways.

Where does the classifier attach? The simplest head flattens the final map into dense layers, and the parameter math quickly gets ugly: flattening 128x4x4 = 2,048 into 10 units is 20,490 weights — acceptable at CIFAR scale, absurd at 512x7x7. The fix (global average pooling) and the deeper question of how many blocks to stack are the two threads pulled in the next section, and the full network you assemble here is the input to Data Augmentation and Transfer Learning.

Why depth? Because features compose. A vertical edge detector is meaningless alone; edges make textures, textures make parts, parts make objects. Early convolutional layers learn generic filters — Gabor-like edges, color opponency — and you can print them: model.layer1[0].conv1.weight visualizes as oriented gratings on any trained network. Middle layers respond to textures and repeated patterns (stripes, fur, brickwork); deep layers respond to parts and finally whole objects, which is why feature visualization papers show neurons that fire on dog faces, car wheels, or text.

VGG's contribution (2014) was to make depth boringly systematic: use only 3x3 convolutions, "same" padding so only pooling reduces size, and double channels after each pool. VGG-16 runs five stages on a 224x224 input: 224 -> 112 -> 56 -> 28 -> 14 -> 7, ending at 7x7x512 before its classifier.

Why 3x3 instead of one 7x7? Two reasons you should be able to produce with a pen:

• Receptive field: two stacked 3x3 layers cover 7x7 (3 + 3 - 1), exactly one 7x7 kernel's view.
• Parameters: with C channels, one 7x7 costs 49 * C^2 weights; two 3x3 cost 2 * 9 * C^2 = 18 * C^2 — 2.7x fewer parameters plus an extra ReLU in between. At C=64: 200,704 vs 73,728 weights.

two_3x3 = nn.Sequential(                       # same 7x7 receptive field
    nn.Conv2d(64, 64, 3, padding=1), nn.ReLU(inplace=True),
    nn.Conv2d(64, 64, 3, padding=1), nn.ReLU(inplace=True),
)
one_7x7 = nn.Conv2d(64, 64, 7, padding=3)

print(sum(p.numel() for p in two_3x3.parameters()))   # 73,856  = 2 * (9*64*64 + 64)
print(one_7x7.weight.numel())                         # 200,704 = 49*64*64

Depth had a ceiling, though, and the observation that defines the era: a 16-layer VGG beats an 8-layer VGG, but a 56-layer plain network trains worse than its 20-layer sibling — higher training error, so not overfitting; the optimization itself degrades with depth. VGG's own parameter profile shows the second problem: of VGG-16's ~138M parameters, roughly 103M sit in one fully connected layer (7x7x512 -> 4096 = 102.7M), because flattening a spatial map into a dense head explodes parameters. Two structural answers fix both issues:

1. Global average pooling replaces the flattening head: average each of the 512 channels to a single number, giving a 512-dim vector and a classifier with 512 * C weights instead of tens of millions.
2. 1x1 convolutions mix channel information at each pixel with no spatial operation — mathematically a shared dense layer applied per position. A 1x1 conv from 512 to 128 channels costs 65,536 weights and lets a network change depth cheaply; stacked with a 3x3 it builds bottleneck blocks.

Those two ideas — cheaper channel mixing and a way to keep 50+ layers trainable — are precisely what ResNet formalized, and that is where the next chapter goes. Two more VGG-era observations are worth carrying: its uniformity (every kernel 3x3, every stage the same recipe) made it the default feature extractor for transfer learning for years — researchers pulled features off a pretrained VGG and ran logistic regression on the output — before lighter backbones such as ResNet-18 displaced it on parameter count alone. And the hierarchy claim is checkable, not folklore: register a forward hook on any trained stage and you will see early maps resolve to oriented edges and color gradients while deep maps activate only around object-sized regions.

Trace the 7x7x512 shape in your head once more: every architectural debate in this topic is a debate about what happens between the input pixels and that final spatial tensor.

ResNet (2015) answered the degradation problem with a one-line change: let stacked layers learn a residual, y = F(x) + x, instead of an exact transformation. If a layer pair is useless, F(x) can collapse to zero and the output equals the input — an identity mapping is trivially available, which a plain stack of convs has to learn the hard way through its whole weight chain.

class BasicBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        out = torch.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return torch.relu(out + x)      # the skip connection

The gradient story is why interviewers love this. Backpropagating through the skip gives dL/dx = dL/dy * (1 + dF/dx) — the +1 term creates a highway where gradient reaches earlier layers unattenuated, regardless of how small dF/dx is. In a plain stack the gradient is a product of 56 matrix-Jacobian terms; if each shrinks the signal by 3%, position 1 receives 0.97^56 ≈ 18% of the gradient, and learning stalls. The skip makes each block responsible for a small correction rather than a full re-derivation. Empirically: ResNet-50 reaches about 76% ImageNet top-1 with 25.6M parameters while VGG-16 sits at ~138M — depth finally stopped being free only in the other direction.

Where 1x1 convolutions enter is the bottleneck block. A naive 256-channel residual block with two 3x3 convs costs 2 * 9 * 256^2 = 1,179,648 weights per block. The bottleneck compresses first:

That is the third key pattern: 1x1 convolutions change channel width (and add nonlinearity) without touching spatial dimensions, which is how bottlenecks, DenseNet-style growth, and the channel mixing in MobileNet's depthwise-separable blocks all work.

The skip itself is not always a literal add — dimensions must match. When a block downsamples (stride 2) or doubles channels (64 -> 128), x cannot be added to F(x) directly, so ResNet routes that shortcut through its own 1x1 convolution with stride 2: y = F(x) + W_s x. At 64 -> 128 the projection costs 64 x 128 = 8,192 weights — the only place skip connections add parameters — and it appears once per stage transition while inner blocks keep the free identity path. Training details finish the picture: the final BatchNorm inside F is zero-initialized, so at step 0 the block computes y = 0 + x exactly, the network starts life as an identity ensemble, and the optimizer begins by learning small corrections instead of digging out of a randomly transformed start. That combination — projection shortcut where dimensions change, identity everywhere else, zero-start residuals — is why a 152-layer ResNet trains without ceremony while a 50-layer plain net fights you.

Note where the accuracy lives — the figure above is architecture alone; real-world accuracy per parameter also depends on the training recipe and data. Data Augmentation and Transfer Learning is about squeezing more signal from the data you have, and it assumes exactly the backbone you can now read off a diagram: conv-ReLU blocks, pooling for downsampling, residuals for depth, and a pooled head for logits.