Why agent benchmark scores depend on the scaffold

Added: 2026-05-25

[00:00:00]Three teams around the same model, same prompt, three different scores. The model was in the variable. When the evaluation lies, the lie above the scaffolding. Welcome back, guys. This is day one, engineering principles for AI agents in prod. The scaffolding is a harness around the model. The what tool agent can call, how many tries it gets, how it tracks its state. Like a query planner around a SQL [music] query. Same statement, different planner, widely different results. On SW bench, the same model can swing up to 13 points or more depending on the harness. The documents spread across all these scaffold research, not [music] a one-off. When you AB test prompts or model, the scaffold is a variable. Lock it or rolling the dice. Common scaffold for the six variable path to checklist. Full breakdown is in the newsletter.