Eval datasets are generated, not found

Entry 7 of the ARENA guide. Part 3.2, dataset generation. Route position: program week 6, tagged [core], the capstone env spec lands this week. Reading thread: Calvano et al. 2020, algorithmic pricing collusion.

The mental model

Last week produced a claim; this week produces the data that could falsify it. The

page's build order teaches the dependency chain by making you climb it: structured

MCQ generation first, then the prompt templates that make generation useful

(few-shot examples, variance prompts so 200 questions are not one question wearing

200 hats), then concurrency so generation scales, then a scoring function, then the

checks and summary statistics that catch a broken dataset before an eval wastes an

API budget on it.

The quiet lesson sits in the ordering: you write the scorer before you mass-produce

the data. A dataset without a scorer is an opinion; a scorer without checks is a

hope. Generation is cheap once the pipeline exists, so the craft goes into the parts

that decide whether the numbers mean anything.

The exercises

Quoted from the pinned 3.2 page, with ARENA's own budgets:

• generate structured MCQ outputs: 10–20 min
• write prompts for question generation: 20–30 min
• write few-shot prompts for generation: 10–20 min
• add variance prompts: 15–20 min
• write a prompt-generation function with concurrency: 10–20 min
• write a scoring function: 25–35 min
• checks and summary statistics: 15–20 min

What it unlocks

The capstone's environment spec v0 lands this week on top of these pieces. Every

later eval (3.3's runs, 3.4's agent tasks, 3.5's control exercises) consumes datasets

built exactly this way, and the collusion literature the reading thread follows lives

or dies on dataset design choices this week makes concrete.

Budget and receipts

ARENA's budgets sum to roughly 1.8–2.7 hours, the lightest week of the core route so

far. Spare slot hours belong to the reading thread: the Calvano paper is long and the

route's collusion anchor.

[TODO: receipts — hours vs budget, dataset size and pass rate, what broke (fill from the TARA vault when the week closes)]

---

PREVIOUS An eval is a threat model with a scoreboard · NEXT Inspect turns an eval into a record you can argue with · GUIDE HOME