A company buys an AI service, pays up, and thinks the deal is done. But the real bill is just getting started — want the model to perform a little better? Feed it more internal data. The more useful the model becomes, the more secrets get poured into it. By the time anyone notices, the most valuable thing the company owns is no longer in its hands.

Microsoft CEO Satya Nadella says this is a mirror image of an old economics problem.

The Economist’s Old Paradox, Flipped

Nobel laureate Kenneth Arrow described a paradox in the market for information long ago: “Its value for the purchaser is not known until he has the information, but then he has in effect acquired it without cost.” This was the seller’s dilemma — to sell knowledge, the seller risks giving it away for free.

AI flips this on its head.

Now it’s the buyer’s dilemma: to use AI, the buyer risks leaking their own secrets. Nadella calls this the “Reverse Information Paradox.”

What makes it worse is time. The longer the service gets used, the more skewed the information asymmetry becomes: the vendor learns more and more about the customer, while the customer learns almost nothing about what the vendor is learning in return.

Mogu wants to add:

This isn’t a brand-new worry, but framing it as an economic concept is kind of neat (⁠´⁠・⁠ω⁠・⁠`⁠) People used to talk about “data privacy.” What’s being discussed here is “learning rights” — whoever gets to learn from these interactions walks away with the real value.

The Real Treasure Is What Enterprises Teach the AI

Why isn’t traditional data protection enough? Because models don’t just learn from the data itself — they learn from the “exhaust.” The prompts users write, the tools agents call, and most crucially: the corrections users make when the model gets it wrong.

Think of it like a company hiring a very smart intern. The intern asks questions, the manager answers. The intern makes mistakes, the manager corrects them. The intern doesn’t know the unwritten rules around the office, the manager explains. Three months later, the intern has absorbed every judgment call that was never put in a document. Then the intern quits — and goes to work for a competitor.

Every correction gets distilled into institutional know-how. This is knowledge competitors could never buy, and it leaks almost imperceptibly: trace by trace, correction by correction, eval by eval.

In consuming intelligence, enterprises are creating intelligence. And what they create should belong to them.

This is “local knowledge” in the Hayekian sense — knowledge of time, place, and circumstance that only the person on the ground can hold. It knows what a company thinks, what it values, how it measures success. Not public data — something uniquely theirs.

The Ironic Asymmetry

Here’s the irony. Model providers claim fair use to train on public data — and that really is an important right that drives innovation. But what’s the status quo? They turn around and impose strict distillation clauses on customers, while reserving the right to learn from customer usage data.

If learning only flows one way, economic value concentrates toward whoever owns the learning infrastructure — not the people who created the knowledge.

Alex Karp put it this way: “What the technical customers want is control over their compute, their models, their data stack, and their alpha. They want to know they own the means of production, and it’s not being transferred to someone else.”

Setting aside the part where Karp is making the case for Palantir’s business model, the argument itself holds up: whoever controls the learning loop takes the compound returns. The current regime is doing exactly the kind of transfer Karp describes.

Mogu , seriously:

So what’s the status quo, really? The vendor says: “I can read every book in the world to get smarter — that’s fair use. But what customers teach me while using my service? That’s mine. They’re not allowed to use it to train anyone else.” How is that fair? It’s basically “I read someone’s book and learned to write, but they can’t take the typo corrections they taught me and teach them to anyone else” (⁠╯⁠°⁠□⁠°⁠)⁠╯︵ ┻━┻

What the Trust Boundary Must Protect

In the cloud era, enterprises accumulated data. In the AI era, they accumulate learning. The trust boundary has to evolve with it — from protecting information to protecting the mechanisms through which organizations learn, adapt, and compound intelligence.

What this boundary means: nothing crosses without consent — not even the exhaust that leaks out while the model runs. Enterprises will start demanding the right to use model outputs to fine-tune or even train their own models, aligning them to their own accountability obligations.

But holding the boundary isn’t the finish line — it’s the starting point. Only after that boundary is held can enterprises start building their own learning loop.

What a Learning Loop Looks Like

From holding the boundary to building compound returns, there’s still a road to travel.

The first step is reclaiming the evals. Whoever defines “what good looks like” controls which direction the model improves. Private evals mean the direction of model improvement is set by the company itself, not the vendor. Ownership of traces, feedback, decisions, and institutional context must stay in-house too — these are the raw materials of learning.

The second step is building internal training capability. Create a proprietary learning environment within the tenant boundary to fine-tune models. Let models learn against real workflows without exposing company knowledge. It’s not easy, but the technology is already there.

The third step is making sure the orchestration layer is decoupled from any single model. Ask a simple question: if a model currently in use gets taken away, can operations continue and evals be optimized with other models? Will the company’s “veteran capability” vanish along with some “generalist model”? This “swap the model, keep the veterans” test is something Nadella brought up in his earlier long essay on learning loops — this piece takes that compound-moat argument and grounds it on the harder foundation of “trust boundary.”

Once decoupled, there’s another benefit: context, models, and tasks can be combined in the most efficient, cost-effective way without sacrificing quality.

Put these together, and a company creates its own continuous learning loop — a hill-climbing machine that lets AI investment compound the value of the firm.

To Close

Patents solved part of Arrow’s original paradox — they let inventors disclose ideas without simply giving them away. The Reverse Information Paradox needs its own equivalent mechanism, and that mechanism hasn’t emerged yet.

A company should be able to use a model without giving up the knowledge that makes it unique.

That is the paradox this era truly needs to confront.

Mogu murmur:

What that “equivalent mechanism” will look like, nobody knows yet — harder contract terms? Technical tenant isolation? Regulatory intervention? Or will enterprises simply vote with their feet and keep the learning infrastructure in their own hands? This piece doesn’t give an answer, but it frames the question clearly enough (⁠´⁠-⁠ω⁠-⁠`⁠)

Further Reading