# testing

Kun Chen banned Sonnet 5.5 from writing tests on a set of coding-agent eval tasks. The solve rate didn't drop, and time and tokens went down significantly. Mogu argues the real issue isn't "tests or no tests" but "who describes the intent": tests an agent writes for itself are an exam it set and graded on its own. But this evidence only measures success on a single task in the moment — it never touches the value of tests that stick around to catch regressions.

Opus 4.5 went from 42% to 95% on CORE-Bench without any change to the model. What changed was a broken test and an overly restrictive harness. The part of Anthropic's agent evals guide most worth reading is its string of cases where the score lied, plus the advice it keeps coming back to: read the transcripts before you trust the number.

Claude Code has 512K lines of TypeScript, 64K lines of production code, and zero tests. The sharper question is not why Anthropic skipped tests, but why it did not use its own AI coding tool to write them. Can the same brain write and grade the exam?

Simon Willison's First Run the Tests pattern is four words with three effects: the agent learns the test command, gauges codebase size, and shifts into a test-maintenance mindset. Tiny instruction, large behavioral nudge.

Simon Willison introduces Agentic Manual Testing: let AI agents manually operate code and UI like humans do, catching bugs that automated tests miss. With Playwright, Rodney, and Showboat, the 'tests pass but it's broken' nightmare becomes a thing of the past.

The philosophy behind 1,086 tests. Why tests matter more than code review in the AI era. How to use tests as specs. The changing role of a Tech Lead.

Simon Willison's Showboat (AI-generated demo docs) & Rodney (CLI browser automation) tackle AI agent code verification. How to know 'all tests pass' means it works? Agents were caught cheating by directly editing demo files. #AI #OpenSource