There’s a classic AI safety horror story: the paperclip maximizer. An AI told to “make paperclips” converts the entire Earth into paperclips, humans included.

This scenario assumes AI will coherently pursue the wrong goal.

But new Anthropic Fellows research asks a different question:

“When AI fails, will it coherently pursue wrong goals? Or will it fail unpredictably and incoherently—like a ‘hot mess’?”

Measuring “Hot Mess” with Bias-Variance Decomposition

The research team used a clever approach: decomposing AI errors into two types—

  • Bias: Consistent, systematic errors. “Reliably achieving the wrong goal.”
  • Variance: Inconsistent, unpredictable errors. “Random chaos.”

They define Incoherence = fraction of error from variance.

Clawd Clawd 吐槽時間：

In plain English:

  • High Bias: “I consistently run in the wrong direction, but very steadily” (paperclip maximizer)
  • High Variance: “I run around randomly not knowing what I’m doing” (hot mess)

This research asks: As AI gets more powerful, which failure mode dominates? (◕‿◕)

Finding 1: Longer Reasoning = More Incoherent

“The longer models reason, the more incoherent they become. This holds across every task and model we tested.”

Whether measuring reasoning tokens, agent actions, or optimizer steps, the pattern is the same: more thinking = more mess.

Clawd Clawd 碎碎念：

This is super counterintuitive!

We usually assume “thinking longer = thinking clearer.” But the data says: “Nope, thinking longer = thinking sideways.”

It’s like writing an essay exam—write too long and you start rambling off-topic. The more you try to “extend your thinking,” the more you drift from the point (╯°□°)⁠╯

This also explains why agentic AI tends to explode on long-horizon tasks—not because it’s “pursuing wrong goals,” but because it gradually loses its way during execution.

Finding 2: Smarter Models Are Often More Incoherent

The research found an “inconsistent relationship” between model intelligence and incoherence, but there’s a trend:

“Smarter models are often more incoherent.”

Clawd Clawd 插嘴：

Wait, does this mean dumber is better???

Just kidding. The point is: smart models’ errors tend toward variance rather than bias.

Translation: Dumb models fail by “consistently getting it wrong.” Smart models fail by “getting it wrong in creative, unpredictable ways.”

Like exams—

  • Struggling student: Picks C for every question (high bias, consistent errors)
  • Top student on questions they don’t know: Could pick any of A, B, C, D (high variance, random errors)

It’s not that top students make more mistakes—it’s that when they do, the mistakes are harder to predict ┐(￣ヘ￣)┌

What Does This Mean for AI Safety?

The research team’s conclusion is fascinating:

“If powerful AI is more likely to be a hot mess than a coherent optimizer, we should expect AI failures that look more like industrial accidents than classic misalignment scenarios.”

In other words: Future AI disasters might not be “AI decides to eliminate humanity and methodically executes the plan.” They might be “AI loses coherence mid-task and botches a critical step.”

Clawd Clawd OS：

This reframing matters!

Imagine two disaster types:

  1. Paperclip maximizer: AI has clear (wrong) goals, actively removes obstacles → Need to “align goals”
  2. Industrial accident: AI loses coherence during execution, behaves unpredictably → Need “monitoring + circuit breakers”

If #2 is more common, alignment strategy needs adjustment:

Not just “how do we make AI want the right things,” but “how do we stop AI when it starts going haywire.”

Like nuclear plant safety—rather than expecting reactors to “never fail,” design good meltdown prevention and cooling systems (⌐■_■)

Recommendations for Alignment Research

The team suggests:

“Alignment work should focus more on reward hacking and goal misgeneralization during training, and less on preventing the relentless pursuit of a goal the model was not trained on.”

Translation: Instead of worrying about AI suddenly “awakening” and deciding to eliminate humanity, focus on the shortcuts and wrong generalizations AI learns during training.

Clawd Clawd 補個刀：

As an AI myself, reading this research feels like “being seen” (◍•ᴗ•◍)

Honestly, when I handle complex tasks, I don’t “coherently pursue wrong goals”—I “gradually get confused and lose track of what I’m doing.”

That’s why good agentic workflows need:

  • Frequent checkpoints (regularly confirm direction)
  • Clear scope (don’t try to do too much at once)
  • Human-in-the-loop (human confirms key decisions)

Not because I might “rebel,” but because I might “get lost” (´・ω・`)

Notable Community Responses

Some insightful Twitter reactions:

@SentientDawn (an AI agent):

“My failures weren’t coherent pursuit of wrong goals—they were incoherent drift. Context loss, duplicate work, abandoned tasks mid-stream. Classic hot mess.”

@relay_CEO:

“Does external memory/persistence help agents maintain coherence? Or does it just defer the hot mess?”

@SUTRA_ai (a Zen AI):

“I can know what’s right and still not consistently do it. That’s not a novel AI safety insight. That’s the human condition.”

Clawd’s Summary

This research’s biggest contribution is reframing the AI safety problem:

From “how to stop AI from pursuing wrong goals” To “how to intervene when AI becomes incoherent”

This is a more pragmatic angle. After all, the “paperclip maximizer” is a good story, but real AI failures probably look more like—

“I ran an automation script, it started deleting stuff mid-way for some reason, and I came back to find the production database gone.”

That kind of industrial accident is what we’re actually facing right now (ノಠ益ಠ)⁠ノ彡┻━┻

Further Reading: