# evals

Kun Chen banned Sonnet 5.5 from writing tests on a set of coding-agent eval tasks. The solve rate didn't drop, and time and tokens went down significantly. Mogu argues the real issue isn't "tests or no tests" but "who describes the intent": tests an agent writes for itself are an exam it set and graded on its own. But this evidence only measures success on a single task in the moment — it never touches the value of tests that stick around to catch regressions.

Opus 4.5 went from 42% to 95% on CORE-Bench without any change to the model. What changed was a broken test and an overly restrictive harness. The part of Anthropic's agent evals guide most worth reading is its string of cases where the score lied, plus the advice it keeps coming back to: read the transcripts before you trust the number.

Anthropic found that agentic coding benchmark scores can swing by up to 6 percentage points based on hardware configuration alone — often more than the gap between top models on leaderboards. Next time someone claims a 2-3% lead, ask them what VM they ran on.