Ten exercises build a transformer around one stream

Entry 4 of the ARENA guide. Part 1.1, transformer from scratch. Route position: program weeks 3–4, tagged [core] (ARENA marks 1.1 + 1.2 compulsory, "at least the first 75%").

The mental model

Strip a GPT-2-style transformer to its skeleton and one shape remains: a stream of vectors running bottom to top, which every component only ever adds to. Embeddings write token identity and position into the stream. Each block reads from the stream, computes something, and adds the result back. Unembed reads the final state as logits. That is the whole architecture. The interpretability slogan later is "the residual stream is the model's workspace, and attention is how information moves between positions in it."

Inside a block there are exactly two workers. Attention lets each position gather from other positions: queries, keys and values, with a causal mask so the past never reads the future. The MLP processes each position alone. LayerNorm keeps the stream's scale sane before each worker reads. ARENA's build order makes the decomposition physical: you implement every box yourself, then train the assembled thing on a tiny corpus and watch it learn.

The picture

Every box is one exercise with ARENA's own budget; the dashed block repeats × N.

The exercises

Ten plus one optional, quoted from the pinned 1.1 page:

• LayerNorm: 10–15 min · Embed: 5–10 min · PosEmbed: 10–15 min
• apply_causal_mask: 10–15 min · Attention: 30–45 min (the hard one)
• MLP: 10–15 min · TransformerBlock: 10–15 min · Unembed: ~10 min
• DemoTransformer: 10–15 min · training loop: 10–20 min
• optional: log completions: 20–40 min

What it unlocks

Chapter 1 is interpretability, and its entire activity is reading the internals of this architecture: probing 1.3's activations, circuit analysis in 1.4, SAEs in 1.5. You cannot hook into what you could not have assembled. The guide's part 0.2 made the same argument for CNNs, and here it becomes the program's premise. The route carries 1.1 §3–4 as [light] in week 4 and treats the full section as a conscious deviation already priced in.

Budget and receipts

ARENA's budgets sum to roughly 2–3 hours, scheduled across weeks 3–4 against the ~4-hour slot. The velocity rule applies on Attention: at 1.5× budget, take the solution. The masking logic is exactly where fiddly debugging pays nothing.

[TODO: receipts: hours vs budget, last exercise reached, training loss curve, what broke (fill from the TARA vault when the weeks close)]

---

PREVIOUS Momentum buys the crossing; plain descent buys the zigzag · NEXT Mech interp reads the activations, not the outputs · GUIDE HOME