ESSAY · PROJECT 0003
Mech interp reads the activations, not the outputs
Entry 5 of the ARENA guide. Part 1.2, intro to mechanistic interpretability. Route position: program weeks 4–5, tagged [core] (§1–2 core; ARENA marks 1.1 + 1.2 compulsory).
The mental model
Part 1.1 ended with a trained transformer. This part asks the question the rest of the program lives on: what are its internals doing? Mechanistic interpretability's bet is that inside the weights there are circuits: identifiable, decomposable machinery, findable with instruments rather than vibes. The instruments are the practical core of §1: hooks and the activation cache let you read any intermediate tensor of the running model; attention visualisation shows which positions gather from which; ablation (deleting a component and measuring the damage) turns a guess about importance into a measurement.
The worked example is the induction head, and it is chosen well: a two-layer circuit (attend to a previous occurrence of the current token, then copy what came after it) that produces a measurable, repeatable behaviour on repeated sequences. §1–2 walk the full arc: predict tokens, inspect activations, visualise attention, write detectors, compute induction scores, attribute logits, ablate heads. After it, "circuit" stops being a metaphor: you have found one, explained one, and broken one on purpose.
The exercises
The spine of §1–2, quoted from the pinned 1.2 page (many are guided walks; the graded ones carry ARENA's budgets):
- inspect your model · how many tokens does your model guess correctly?: ~10 min
- verify activations: 10–15 min · visualise & inspect attention patterns: ~10 min
- write your own detectors · per-token loss on a repeated sequence
- make an induction-head detector · calculate induction scores with hooks
- find induction heads in GPT-2-small · logit attribution tool
- interpret the logit attribution · induction-head ablation · mean ablation
- interpret the logit attribution · induction-head ablation · mean ablation
Section §3 (TransformerLens: hooks) stays in route for week 5: HookedTransformerConfig, hook functions, and running with hooks, the instrumentation the W40 threat-model week assumes you can already wire.
What it unlocks
Everything downstream interprets this part's toolkit. 1.3's probes read the same activations; 1.4's circuit analysis is the attribution-and-ablation loop at scale; the capstone's evaluation work leans on the same instrumentation habit. The route puts 1.2 §1–2 in week 5, right after the transformer exists. ARENA's compulsory pair, learned back to back.
Budget and receipts
§1–2 are guided-plus-graded walks with the budgets quoted above; the full 1.2 part runs deep into circuits the route defers (OV and QK circuits are §2's tail end). The plan's hour cap is the ~4-hour weekly slot; the velocity rule applies to the detector exercises.
[TODO: receipts: hours vs budget, induction-head detector results, ablation deltas, what broke (fill from the TARA vault when the weeks close)]