ESSAY · PROJECT 0003
You build ResNet34 by hand before anyone hands you a model
Entry 2 of the ARENA guide. Part 0.2, CNNs & ResNets. Route position: program week 2, tagged [core], the first TARA session's material.
The mental model
A convolution is a window that walks. The kernel is a small grid of weights, the same weights at every position, and the output is what the kernel sees summed position by position. Two consequences fall out of that one sentence. Weight sharing is why a convolutional layer needs thousands of parameters where a dense layer on the same image needs millions; the TARA session deck puts the blow-up at 154M parameters without sharing. And locality is a prior, not a trick: nearby pixels are relevant to each other, distant ones reach each other only through layers of composition.
Pooling is the vote that follows the window: max or average, shrinking the map while keeping what the kernels found. And a residual block is the answer to the problem depth creates: gradients fading on their way back through twenty layers. The skip connection hands the gradient a bypass. The residual stream (the thing transformers later reuse verbatim) is memory that runs up the network, and every block only ever adds to it.
The picture
Step through the nine windows. Same kernel every step, one output cell per position. That is the whole operation.
The exercises
Eleven, quoted from the pinned 0.2 page with ARENA's own budgets:
ReLU: ~10 min ·Linear: ~10 min · simple MLP: ~20 min- validation loop: ~20 min · train the MLP on MNIST
Conv2d: 10–20 min ·BatchNorm2d: 15–30 min ·AveragePool: 5–10 minResidualBlock: 20–30 min ·BlockGroup: 10–15 minResNet34: 30–45 min (ARENA warns: "a lot of fiddly debugging") · verify predictions: ~10 min
What it unlocks
The route's plan note is explicit: 1.1 (transformer from scratch) needs the nn.Module and training-loop fluency this part builds. Interpretability later is reading the internals of architectures you can build from scratch. You cannot hook into what you could not have assembled. The session deck's four takeaways end on the bridge this part builds: the residual stream is the exact architecture idea transformers reuse.
Budget and receipts
ARENA's budgets sum to roughly 3–3.5 hours of exercises, inside the week's ~4-hour slot. The velocity rule applies hard on ResNet34: at 1.5× budget, take the solution and move on.
[TODO: receipts: hours vs budget, MNIST test accuracy, ResNet34 prediction check, what broke (fill from the TARA vault when the week closes)]