Eval datasets are generated, not found

Entry 7 of the ARENA guide. Part 3.2, dataset generation.

The mental model

The previous part produced a claim; this part produces the data that could falsify it.

The page's build order teaches the dependency chain by making you climb it: structured

MCQ generation first, then the prompt templates that make generation useful

(few-shot examples, variance prompts so 200 questions are not one question wearing

200 hats), then concurrency so generation scales, then a scoring function, then the

checks and summary statistics that catch a broken dataset before an eval wastes an

API budget on it.

The quiet lesson sits in the ordering: you write the scorer before you mass-produce

the data. A dataset without a scorer is an opinion; a scorer without checks is a

hope. Generation is cheap once the pipeline exists, so the craft goes into the parts

that decide whether the numbers mean anything.

If you want to build this

Quoted from ARENA’s 3.2 page, with its own timed budgets; they sum to roughly 1.8–2.7 hours, the lightest set in this guide:

• generate structured MCQ outputs: 10–20 min
• write prompts for question generation: 20–30 min
• write few-shot prompts for generation: 10–20 min
• add variance prompts: 15–20 min
• write a prompt-generation function with concurrency: 10–20 min
• write a scoring function: 25–35 min
• checks and summary statistics: 15–20 min

What it unlocks

Every later part (3.3's runs, 3.4's agent tasks, 3.5's control exercises) consumes

datasets built exactly this way, and the collusion literature this guide pairs with the

part lives or dies on dataset design choices these exercises make concrete.

Go deeper

• The ARENA 3.0 curriculum, chapter 3, part 3.2.
• The founding paper on algorithmic collusion: Calvano et al. (2020), “Artificial Intelligence, Algorithmic Pricing, and Collusion”.

---

PREVIOUS An eval is a threat model with a scoreboard · NEXT Inspect turns an eval into a record you can argue with · GUIDE HOME