ESSAY · PROJECT 0003
Reasoning models say more than they do, so measure the gap
Entry 13 of the ARENA guide. Part 4.3 §1–2, interpreting reasoning models.
The mental model
Reasoning models produce a chain of thought, and safety work leans on it: if the
model says why it did something, a monitor can read that. The uncomfortable question
this part formalizes: is the chain of thought faithful to the computation that
produced the answer, or is it a plausible story told after the fact? "Interpreting
reasoning models" turns that worry into measurement.
The exercises build the measurement in layers. Categorize reasoning sentences
heuristically, then with an autorater. Then the core idea, importance: how much
does the final answer depend on a given piece of reasoning? Forced-answer importance
tests it directly (truncate the reasoning, force an answer, see if it changes);
resampling and counterfactual importance statisticalize it. The payoff exercise
replicates a published figure on how sentence categories in the chain relate to
outcomes. The habit being installed: never quote a model's self-report as evidence
without a faithfulness number attached.
If you want to build this
Quoted from ARENA’s 4.3 page, with its own timed budgets; they sum to roughly 1–1.5 hours, deliberately light. No GPU appears anywhere:
- heuristic-based categorization: 10–15 min
- implement an autorater: 10–15 min
- calculate forced answer importance: 10–15 min
- compare resampling importance: 5–10 min
- compute counterfactual importance: 15–20 min
- replicate Figure 3b (sentence category effect): 10–15 min
What it unlocks
This closes the guide's core route. Chain-of-thought monitorability is exactly the
kind of claim a serious evaluation makes about itself, and with this part the guide
has covered the whole arc: build the model, read the model, test the model, control
the model, and measure the model's honesty about itself.
Go deeper
- The ARENA 3.0 curriculum, chapter 4, part 4.3 §1–2.
- Meek et al., on measuring chain-of-thought monitorability.