ESSAY · PROJECT 0003

Ten exercises build a transformer around one stream

Dataset the ARENA 3.0 curriculum (TARA vault submodule, pinned 81533b2) + the TARA Learning Plan v2 · Charts 1 · Written SEP 2026

Entry 4 of the ARENA guide. Part 1.1, transformer from scratch.

The mental model

Strip a GPT-2-style transformer to its skeleton and one shape remains: a stream of vectors running bottom to top, which every component only ever adds to. Embeddings write token identity and position into the stream. Each block reads from the stream, computes something, and adds the result back. Unembed reads the final state as logits. That is the whole architecture. The interpretability slogan later is "the residual stream is the model's workspace, and attention is how information moves between positions in it."

Inside a block there are exactly two workers. Attention lets each position gather from other positions: queries, keys and values, with a causal mask so the past never reads the future. The MLP processes each position alone. LayerNorm keeps the stream's scale sane before each worker reads. ARENA's build order makes the decomposition physical: you implement every box yourself, then train the assembled thing on a tiny corpus and watch it learn.

The picture

Every box is one exercise with ARENA's own budget; the dashed block repeats × N.

If you want to build this

Ten plus one optional, quoted from ARENA’s 1.1 page; the budgets sum to roughly 2–3 hours:

LayerNorm: 10–15 min · Embed: 5–10 min · PosEmbed: 10–15 min
apply_causal_mask: 10–15 min · Attention: 30–45 min (the hard one)
MLP: 10–15 min · TransformerBlock: 10–15 min · Unembed: ~10 min
DemoTransformer: 10–15 min · training loop: 10–20 min
optional: log completions: 20–40 min

The masking logic is exactly where fiddly debugging pays nothing.

What it unlocks

Chapter 1 is interpretability, and its entire activity is reading the internals of this architecture: probing 1.3's activations, circuit analysis in 1.4, SAEs in 1.5. You cannot hook into what you could not have assembled. The guide's CNN entry made the same argument for images, and here it becomes the premise of everything chapter 1 does.

Go deeper

The ARENA 3.0 curriculum, chapter 1, part 1.1 — build the ten exercises yourself.

PREVIOUS Momentum buys the crossing; plain descent buys the zigzag · NEXT Mech interp reads the activations, not the outputs · GUIDE HOME