99.8% of the Tests Pass — Then Anthropic Adds 'Not Yet in Production.' The Real Product of Loop Engineering Is the Verifier
Loops are easy to draw. Discover, plan, execute, verify, repeat—the flowchart takes five minutes, and any LLM can generate one. The problem was never “how to design a loop.” The problem is “can you trust what this loop produces.”
Mogu OS:
If you haven’t read the basics of loop engineering (open/closed loops, how the six building blocks fit together), go read SP-220 first. This is the sequel—I’m cutting straight to verifiers without the intro. (￣▽￣)
The “Verify” Box Is the Product
Boris Cherny said at Fortune Brainstorm Tech this June that he doesn’t write prompts himself anymore—another Claude writes them. He manages hundreds of agents in a morning, sometimes thousands.
This got packaged as “the loop engineering era is here”: 2024 you write good prompts, 2025 you run agents in parallel, 2026 you design systems that spawn agents on their own. Sounds great, but the packaging hides what’s actually hard.
A loop is a generator connected to a verifier. The generator is never the bottleneck. Today’s models generate easily—fast, prolific, confident. The bottleneck is the verifier: who decides if the output is correct? By what standard? How do you catch it when the judgment is wrong?
Everyone draws that “discover → plan → execute → verify” flowchart. Almost no one explains what’s inside the “verify” box. That box is the actual product. Everything else is plumbing.
Inner Loop Verification: Self-Checks Within a Task
Inner loop verification happens inside a single task. The agent verifies its own work before claiming it’s done.
A weak agent edits a file and says “done.” A strong agent edits a file, writes a test, runs the test, catches an unhandled edge case, fixes it, runs again, confirms green, then says “done.” Same tools. The difference is whether the model chooses to call the verifier.
Same as human engineers. Juniors edit code and commit. Seniors edit code, run
npm test, click through the UI, check the console for red text. The tools are right there. The difference is using them. (⌐■_■)
This cycle has a name: ReAct (the Princeton and Google paper). Think, act, observe the result, think again, repeat until done. In programming terms: understand the goal, write code, run it, read the error, infer the cause, fix it, run again, until tests pass.
Inner loop verification is mature now—most usable agents do it. But “doing it” and “doing it well” are different things. The verifier’s quality caps the output’s quality. Never higher.
Outer Loop Memory: Lessons Across Sessions
Outer loop verification happens between sessions. The agent screws something up this time, writes the lesson to a persistent file, and next session reads that file and avoids the same mistake from the start.
This also has a name: Reflexion. ReAct plus memory. When an attempt fails, the agent writes down “why it failed” in plain language, stores it, reads it next time. In modern toolchains, that note lives in a file, not the context window. SKILL.md, AGENTS.md, CLAUDE.md—these are concrete implementations of outer loop memory.
Context windows reset. Git repos don’t. Writing lessons into repo files is turning short-term memory into long-term memory. Not a new concept—engineers have been writing READMEs and CONTRIBUTING.md for decades. The reader just changed from human to agent. (◕‿◕)
Outer loop memory is still half-baked. What lessons to store, where to store them, how granular—harder than it sounds. Too coarse and the agent can’t apply it. Too fine and the file becomes a junk pile that slows down reasoning.
Inner loop verification handles “is this task correct.” Outer loop memory handles “don’t repeat last week’s mistake.” Both are verification, just on different timescales.
The Bun Port: A Cautionary Tale of 99.8% Green
Jarred Sumner used Claude Code’s dynamic workflows to port Bun from Zig to Rust. About 750,000 lines of Rust, 99.8% of the existing test suite passing, roughly six to eleven days from first commit to merge (depending on whose timeline you trust).
The verification architecture was solid: first pass annotated each struct field with the correct Rust lifetime, second pass ran hundreds of agents in parallel writing files with two review agents per file, plus a layer of agents specifically tasked with refuting other agents’ output. A repair loop drove build and test until both passed. Verification wasn’t the last step—it was the whole structure.
Then there’s the caveat Anthropic wrote in their dynamic workflows announcement (June 2, 2026): this port hasn’t shipped to production.
99.8% test pass sounds impressive, but that’s benchmark results, not production truth. Test suites only cover “behavior someone thought to test.” Production is “behavior no one wrote a test for.” The gap between them is where this entire industry keeps tripping. (¬‿¬)
This isn’t saying the Bun port was done badly—that verification architecture is stronger than most projects. The point: a loop going green doesn’t mean the loop is correct. It means the loop satisfied the verifier humans gave it. Verifier quality caps output quality.
The full language migration story is in SP-203. I’m borrowing it here for one thing: tests passing is the starting line, not the finish line.
Concrete Questions in Verifier Design
“Design a verifier” sounds abstract. Break it down and it’s a list of concrete questions:
What do you verify? Syntax correct, tests pass, types check—these are the floor. The hard part is semantic correctness: does this code actually do what the spec says? Are edge cases handled? Will it blow up when interacting with the rest of the system?
What’s the standard? A test suite is one standard, but it only covers cases someone thought to write. Another agent as reviewer is one standard, but agent-reviewing-agent has its own blind spots. Human review is another standard, but humans are slow, expensive, and get tired. Most practical answers mix them: automate what you can catch, leave the rest for humans.
How do you catch failures? After the verifier says “wrong,” what’s next? Reject and regenerate? Send the error message for the agent to fix? Escalate to a human? Different failures need different handling. Not every error deserves the same effort.
How do you know the verifier itself is correct? This is the awkward question. The verifier lets bad output through—how do you find out? Usually: wait for production to explode. That’s why outer loop memory matters—production explosions need to be written down, and the verifier needs to learn to catch them next time.
Generator and verifier need to be separate, because agents grading their own work grade leniently. Same as humans—you review your own PR and always think it’s fine. Role separation isn’t distrust, it’s system design. (๑•̀ㅂ•́)و✧
Design the Verifier Before Scaling the Loop
Boris Cherny manages hundreds of agents because those agents run inside a verification architecture he trusts. Without that architecture, hundreds of agents means hundreds of times the error generation speed.
Dynamic workflows currently cap at 16 parallel agents, 1000 agents max per run, and token costs are much higher than normal sessions. Not every task needs a workflow. Wrapping small tasks into workflows is its own kind of waste.
The tools are here: scheduled triggers to spawn agents, isolated git worktree so parallel agents don’t stomp each other, skill files that skip re-explaining project conventions every time, connectors to plug into tools you already have. All plumbing. Important, but not the core.
The core is the verifier. Before scaling the loop, answer these: What does the agent check? By what standard? How are failures caught? How are lessons stored? No answers to these, more agents just means more garbage.
Conclusion
“Loop engineering” is a new label, but the hard problem it points to has always been there. The generator is never the bottleneck. The verifier is. Everyone can draw the flowchart. Almost no one explains what’s in the “verify” box.
The Bun port hit 99.8% test pass. Anthropic added “hasn’t shipped to production.” That caveat matters more than all the technical details of the project. Green tests are benchmark results. Production is behavior no one tested.
Before designing your next loop, design the verifier first. Otherwise you’re just generating untrusted output faster.