ESSAY · PROJECT 0003
Momentum buys the crossing; plain descent buys the zigzag
Entry 3 of the ARENA guide. Part 0.3 §1, optimization. Route position: program week 3, tagged [light] (§1 only; §2 Weights & Biases and §3 distributed training sit outside the core route).
The mental model
Gradient descent on a canyon-shaped loss surface has a pathology worth seeing before trusting any trainer. When one axis is steep and the other is flat, the step size that makes progress on the flat axis overshoots the steep one. Descent zigzags across the canyon walls, losing most of its motion to the direction it did not want to travel.
Momentum changes what a step is. Instead of a blind footstep proportional to the current gradient, the update carries velocity: an exponentially weighted memory of past gradients. The zigzag components point in alternating directions and cancel inside that memory; the canyon direction agrees with itself every step and compounds. Damped oscillation on the steep axis, acceleration on the flat one. Same learning rate, different walk. Adaptive methods (RMSProp, Adam) attack the same pathology from the other side, rescaling each coordinate by its own gradient history.
The picture
Both walkers start in the same ring, see the same gradients, and take 24 steps. The gray path is plain descent; the ink path carries momentum. The math is real; only the terrain is illustrative.
The exercises
Four, quoted from the pinned 0.3 page with ARENA's own budgets:
opt_fn_with_sgd: 15–20 min (wrap an existing optimizer for the visualization)- implement
SGD: 25–35 min - implement
RMSprop: 15–25 min - implement
Adam: 15–20 min
What it unlocks
The route tags this section [light] because the training-loop habit already exists from 0.2. What is new here is the optimizer's interior. You will trust Adam as a black box in 1.1's training exercise; four exercises buy the right to. ARENA's own visualization section ("pathological curvatures") is where this entry's canyon picture comes from.
Budget and receipts
ARENA's budgets sum to roughly 70–100 minutes for the four exercises. The plan schedules 0.3 §1 in week 3 alongside the start of 1.1, and moves the rest of 0.3 out of the core route: distributed training and the W&B dashboard are real skills, but none of the capstone's pages depend on them.
[TODO: receipts: hours vs budget, last exercise reached, what broke (fill from the TARA vault when the week closes)]