ESSAY · PROJECT 0003
Momentum buys the crossing; plain descent buys the zigzag
Entry 3 of the ARENA guide. Part 0.3 §1, optimization: the optimizers themselves.
The mental model
Gradient descent on a canyon-shaped loss surface has a pathology worth seeing before trusting any trainer. When one axis is steep and the other is flat, the step size that makes progress on the flat axis overshoots the steep one. Descent zigzags across the canyon walls, losing most of its motion to the direction it did not want to travel.
Momentum changes what a step is. Instead of a blind footstep proportional to the current gradient, the update carries velocity: an exponentially weighted memory of past gradients. The zigzag components point in alternating directions and cancel inside that memory; the canyon direction agrees with itself every step and compounds. Damped oscillation on the steep axis, acceleration on the flat one. Same learning rate, different walk. Adaptive methods (RMSProp, Adam) attack the same pathology from the other side, rescaling each coordinate by its own gradient history.
The picture
Both walkers start in the same ring, see the same gradients, and take 24 steps. The gray path is plain descent; the ink path carries momentum. The math is real; only the terrain is illustrative.
If you want to build this
Four exercises, quoted from ARENA’s 0.3 page with its own timed budgets; they sum to roughly 70–100 minutes:
opt_fn_with_sgd: 15–20 min (wrap an existing optimizer for the visualization)- implement
SGD: 25–35 min - implement
RMSprop: 15–25 min - implement
Adam: 15–20 min
What it unlocks
The training-loop habit already exists from the CNN part; what is new here is the optimizer's interior. You will trust Adam as a black box in the transformer's training exercise, and these four exercises buy the right to. ARENA's own visualization section (“pathological curvatures”) is where this entry's canyon picture comes from. The rest of 0.3 — distributed training and the Weights & Biases dashboard — is real skill, but nothing later in this guide depends on it, so the entry stops at the optimizers.
Go deeper
- The ARENA 3.0 curriculum, chapter 0, part 0.3 §1 — including the “pathological curvatures” visualizations this entry’s canyon comes from.