evals
5 articles
A Practical Guide to Agent Evals: Don’t Just Look at What It Says—Look at What the World Becomes
Agent evals must verify not only the response, but whether the world actually changed. Anthropic shows how to combine graders, isolate environments, and maintain an eval suite over time—and why 20–50 real failures are enough to start.
The Hard Part of Agents Is Not the Model. It Is the Engineering Floor.
A practical agent engineering guide covering control loops, harnesses, context engineering, tool design, memory, multi-agent systems, evals, tracing, and safety boundaries.
Skillify: Turn Every Agent Failure Into Something Structurally Impossible to Repeat — Garry Tan's 10-Step Checklist
Garry Tan's agent screwed up twice this week — both bugs had the same shape: deterministic work done in latent space. His fix is skillify: every failure becomes a SKILL.md + deterministic script + tests + evals + resolver trigger. Ten steps. The bug becomes structurally impossible to repeat.
Eval-Driven Development — You Test Your Code, But Who Tests Your AI?
You use unit tests to check your code and CI to protect your pipeline. But who checks your AI? Eval-Driven Development (EDD) upgrades AI development from "looks good to me" to actual engineering — with pass@k metrics, three grader types, and product vs regression evals. This is TDD for the AI era.
Anthropic Exposes AI Benchmarks' Dirty Secret — Leaderboard Gaps Might Just Mean 'Bigger VM'
Anthropic found that agentic coding benchmark scores can swing by up to 6 percentage points based on hardware configuration alone — often more than the gap between top models on leaderboards. Next time someone claims a 2-3% lead, ask them what VM they ran on.