Inspect turns an eval into a record you can argue with

Entry 8 of the ARENA guide. Part 3.3, running evals with Inspect.

The mental model

The previous two parts built eval pieces by hand; here the pieces meet a framework.

inspect_ai is the UK AI Safety Institute's evaluation framework, and ARENA's build

order maps its vocabulary onto things you already own: a dataset becomes Sample

objects (via your record_to_sample function), the eval logic becomes a solver

(prompt templates, multiple-choice formatting, self-critique), the grading becomes a

scorer, and the whole run becomes a structured log you can reopen and argue with.

The concept worth fighting for: solvers are composable. A solver wraps the model call,

so formatting, critique passes and tool use are all just solvers stacked in a

pipeline. Once that clicks, "running an eval" stops being a script and becomes a

configuration, which is what any real evaluation project needs once the scaffolding

gets complicated.

If you want to build this

Quoted from ARENA’s 3.3 page, with its own timed budgets; they sum to roughly 1–2 hours, the shortest set in this guide:

• write record_to_sample for your dataset: 10–15 min
• implement prompt_template solver: 10–25 min
• implement multiple_choice_format solver: 10–20 min
• implement make_choice solver: 5–10 min
• implement self_critique_format solver: 10–20 min
• write a simple scorer · (optional) your own solvers and scorers: 15–25 min

What it unlocks

Everything downstream runs on this framework: the agents and control protocols of the

next two parts are Inspect tasks. This is also where the phrase "the log is the

deliverable" becomes practical, because every claim a project makes later has to

survive someone reopening its logs.

Go deeper

• The ARENA 3.0 curriculum, chapter 3, part 3.3, and the inspect_ai documentation.
• Korbak et al., on evaluating control measures for LLM agents.

---

PREVIOUS Eval datasets are generated, not found · NEXT An agent is a loop, not a model · GUIDE HOME