Same model, same reasoning strength—swap only the Agent Harness that calls it—and on some cases the per-task cost more than doubles while quality stays put. Picture the same kitchen and the same cooks; only the plating workflow changes. Some people haul half the fridge to the table with every dish; others bring only what that dish actually needs. Of course the bill looks different. In a controlled experiment from a cloud data-platform team, Pi sent about one-third as much context per turn as the comparison baseline, and finished tasks in fewer turns; both show up alongside the lower per-task cost.

These numbers split the coding-agent cost problem open. The model price list is just the unit rate. How much stale baggage the harness carries each turn, and how many loops it makes the model take, is what settles the final bill.

Mogu wants to add:

The source is from Pi’s vendor—discount the ranking and the marketing tone. Background reading if you want it: Pi’s minimal design, the harness is the product, context cost, the Autoresearch loop, and an e-commerce platform case. This piece only covers the new controlled experiment. (⁠¬⁠‿⁠¬⁠)

Why these numbers beat a leaderboard

Public problem sets are like a school-wide exam: everyone has seen the questions, and a high score does not mean you can do the job at work. The research team did not grind that set again. They built tasks from real changes in their multi-million-line codebase—filtering out bot and fully auto-generated commits, keeping cases with reliable tests, then human-checking questions and answers. That looks more like the tickets engineers actually hit every day, and it cuts the risk that public answers leaked into training data.

More important: control the variables. Once model and reasoning strength are fixed, the price gap is easier to pin on how the harness shapes the working set, resends context, and ends the task. The result does not mean one harness is always cheaper, and it does not mean native tooling is always worse. It only shows that “model name” is not a complete unit of cost—like staring at the electricity rate while ignoring how many hours the AC runs.

Where the cost leaks

Pi defaults to four tools; system prompt plus tool definitions stay under 1,000 tokens. Extend when something is missing—no need to make every task read a full rulebook it will never use. Imagine you only need to tighten one screw, but someone carts over the entire hardware store “just in case.” That is not caution; that is freight written into the quote. The smaller working sets and fewer turns observed in the study turn this design from a “minimalist preference” into a measurable cost gap.

So price per million tokens alone does not stand in for engineering cost. Pass rate per task, total cost per task, context per turn, and total turns have to be compared together.

A thin harness can still grow heavy workflows

The next doubt lands here: if the default is this thin, does capability get cut too? An engineering write-up from an e-commerce platform describes how they call Pi, via extension docs, to run Autoresearch: each round measure the baseline, propose a hypothesis, keep improvements, discard regressions, then keep experimenting. The chassis stays light; heavy workflow hangs off extensions instead of stuffing the whole process into the system prompt at boot.

Cases include unit tests sped up 300×, React component mount ~20% faster, build times down across multiple projects, and pnpm performance gains. These are company cases, not a generalizable public benchmark. The narrow claim they support: a thin default and a heavy workflow are not in conflict—if the extension points are good enough to use.

Turn complexity into measurable cost

Once model and reasoning strength are fixed, comparing harnesses on pass rate, total cost, context per turn, and total turns is what separates “looks stronger” from “costs more on the bill.” Complexity can be added—but only after you can measure, on your own tasks, whether it is worth it.

Mogu 's hot take:

Complexity is allowed—it just has to pay rent. Defaults that never prove their usefulness in numbers will show up on the bill.

If you actually use this to pick tools: pull real tasks from your own merge history, score success with existing tests, then fix model and reasoning strength and compare harnesses. Other people’s leaderboards are fine as a starting point—not as a purchase answer. (⁠・⁠ω⁠・⁠)