Inspect turns an eval into a record you can argue with

Entry 8 of the ARENA guide. Part 3.3, running evals with Inspect. Route position: program week 7, tagged [core]. Reading thread: Korbak et al., evaluating control measures for LLM agents.

The mental model

Weeks 5 and 6 built eval pieces by hand; this week the pieces meet a framework.

inspect_ai is the UK AI Safety Institute's evaluation framework, and ARENA's build

order maps its vocabulary onto things you already own: a dataset becomes Sample

objects (via your record_to_sample function), the eval logic becomes a solver

(prompt templates, multiple-choice formatting, self-critique), the grading becomes a

scorer, and the whole run becomes a structured log you can reopen and argue with.

The concept worth fighting for: solvers are composable. A solver wraps the model call,

so formatting, critique passes and tool use are all just solvers stacked in a

pipeline. Once that clicks, "running an eval" stops being a script and becomes a

configuration, which is what the capstone will need when the scaffolding gets

complicated.

The exercises

Quoted from the pinned 3.3 page, with ARENA's own budgets:

• write record_to_sample for your dataset: 10–15 min
• implement prompt_template solver: 10–25 min
• implement multiple_choice_format solver: 10–20 min
• implement make_choice solver: 5–10 min
• implement self_critique_format solver: 10–20 min
• write a simple scorer · (optional) your own solvers and scorers: 15–25 min

What it unlocks

The capstone runs on this framework: 3.4's agents and 3.5's control protocols are

inspect tasks, and the route's W44 gate expects fluency here. This is also the week

the phrase "the log is the deliverable" becomes practical, because every claim the

capstone makes later has to survive someone reopening its logs.

Budget and receipts

ARENA's budgets sum to roughly 1–2 hours, the shortest core week so far. The route

spends the slack on 3.4's reading and the Korbak paper, both heavier than their slot.

[TODO: receipts — hours vs budget, eval run results, log inspection notes, what broke (fill from the TARA vault when the week closes)]

---

PREVIOUS Eval datasets are generated, not found · NEXT An agent is a loop, not a model · GUIDE HOME