ESSAY · PROJECT 0003
Mech interp reads the activations, not the outputs
Entry 5 of the ARENA guide. Part 1.2, intro to mechanistic interpretability.
The mental model
Part 1.1 ended with a trained transformer. This part asks the question the rest of the program lives on: what are its internals doing? Mechanistic interpretability's bet is that inside the weights there are circuits: identifiable, decomposable machinery, findable with instruments rather than vibes. The instruments are the practical core of §1: hooks and the activation cache let you read any intermediate tensor of the running model; attention visualisation shows which positions gather from which; ablation (deleting a component and measuring the damage) turns a guess about importance into a measurement.
The worked example is the induction head, and it is chosen well: a two-layer circuit (attend to a previous occurrence of the current token, then copy what came after it) that produces a measurable, repeatable behaviour on repeated sequences. §1–2 walk the full arc: predict tokens, inspect activations, visualise attention, write detectors, compute induction scores, attribute logits, ablate heads. After it, "circuit" stops being a metaphor: you have found one, explained one, and broken one on purpose.
If you want to build this
The spine of §1–2, quoted from ARENA’s 1.2 page (many are guided walks; the graded ones carry ARENA's own budgets):
- inspect your model · how many tokens does your model guess correctly?: ~10 min
- verify activations: 10–15 min · visualise & inspect attention patterns: ~10 min
- write your own detectors · per-token loss on a repeated sequence
- make an induction-head detector · calculate induction scores with hooks
- find induction heads in GPT-2-small · logit attribution tool
- interpret the logit attribution · induction-head ablation · mean ablation
Section §3 (TransformerLens: hooks) continues the toolkit: HookedTransformerConfig, hook functions, and running with hooks — the instrumentation the evals chapter assumes you can already wire.
What it unlocks
Everything downstream interprets this part's toolkit: 1.3's probes read the same activations, 1.4's circuit analysis is the attribution-and-ablation loop at scale, and any evaluation project you build leans on the same instrumentation habit. It is learned right after the transformer exists — ARENA's compulsory pair, back to back.
Go deeper
- The ARENA 3.0 curriculum, chapter 1, part 1.2 — hooks, attention patterns, attribution and ablation.