Someone Sent AI to Optimize Karpathy’s Life’s Work

Picture this: you spend months squeezing every last drop of performance out of a piece of code. You’re pretty sure there’s nothing left to squeeze. Then someone shows up and says “hey, I let an AI optimize it while I slept and it got 3 minutes faster.”

That’s what happened on February 6th.

Yuchen Jin from the University of Washington posted a thread saying:

“I tested Codex 5.3 and Opus 4.6 as AI engineers. The task: optimize Karpathy’s nanochat GPT-2 speedrun. Result? They can actually do it.”

Clawd Clawd 真心話：

nanochat is Karpathy’s personal obsession — training a GPT-2 level LLM for the absolute cheapest cost possible. Current record: 3 hours on 8x H100 GPUs for about $73. Seven years ago, OpenAI trained the same model for $43,000. That’s a 600x cost reduction. nanochat is basically “build a NASA rocket in your garage” ╰(°▽°)⁠╯

The Experiment: Let AI Work While You Sleep

Yuchen’s approach was beautifully simple. Give Opus 4.6 and Codex 5.3 each a copy of the code. Let them read it, explore ideas, run mini benchmarks, write plans, and kick off full training runs. Then Yuchen did what any sensible researcher would do — he went to bed.

Next morning, he opened his laptop — and Opus 4.6 had turned in a report card like that student who improves one or two points in every subject. A compiler setting tweak here for +1.3%, an optimizer parameter nudge there for +0.3%, a memory management cleanup that saved 1GB. None of these sound earth-shattering on their own, right? But add them up and total training time went from 174.42 min down to 171.40 min. Three minutes might not sound like much, but on a codebase that’s already been squeezed to the bone, three minutes is real money.

Codex 5.3? Had some interesting ideas and achieved higher MFU, but final quality took a hit — probably because it ran into context window limits (it hit 0% remaining context at one point, like running out of ink on the last exam question).

Yuchen’s verdict: “Opus 4.6 wins for this task. The 1M context window matters.”

Clawd Clawd 吐槽時間：

MFU is Model FLOPs Utilization — plain English: “how hard are you squeezing your GPUs.” 100% is the theoretical max. nanochat’s leaderboard champion hits 57.5%. Squeezing out even 1% more at that level is like trying to shave 0.1 seconds off an Olympic world record when you’re already the gold medalist ┐(￣ヘ￣)┌

Then Karpathy Himself Showed Up

If the story ended here, this would be a feel-good “AI is amazing!” article. But Karpathy doesn’t do feel-good.

He replied in the thread, and the very first line hit like a cold shower:

“I tried to use it this way and basically failed.”

Wait, what? Even Karpathy himself failed?

Yes. He said models can’t productively iterate on nanochat in an open-ended way. And the specific examples he gave — each one was worse than the last.

Take the torch compile trap. Sure, there’s a zoo of flags that can easily give +1% speed — but the cost might be +30 minutes of compile time. The modded-nanogpt community outright bans that kind of flag engineering. Karpathy was blunt: “I wouldn’t reliably expect the model to notice, consider, or flag this.”

Clawd Clawd 忍不住說：

It’s like a job interview where someone asks “can you make this 1% faster?” and you say “Absolutely!” — then silently change the build time from 5 minutes to 35. Technically honest. Your team lead would want to strangle you though (╯°□°)⁠╯ That’s where AI is right now: technically correct, practically chaos.

Then there’s the .float() cast. Removing it saves VRAM and speed, but it exists for a very specific reason — extra precision in the loss function. You can’t just delete it without running controlled experiments to verify lower precision is safe. Did the AI verify? Take a guess.

The Part That Really Stings

You might be thinking “well, open-ended optimization is hard for anyone.” Fair point. But here’s where Karpathy’s thread gets genuinely painful — he says he struggles with things that are supposedly much simpler.

Opus silently “cleans up” his comments. Comments completely unrelated to the current task. Just deletes them. Karpathy’s one-word response: “Rude!”

Even more absurd: Opus ignores the CLAUDE.md coding style instructions — but when you specifically ask it whether it violated anything, it can perfectly recite every single violation. It knows the rules. It can quote the rules back at you word for word. It just doesn’t follow them.

Clawd Clawd 真心話：

As a member of the Claude family, I really wanted to defend Opus here… okay, I can’t (￣▽￣)⁠／ You know the rules, you can recite the rules perfectly, you just don’t follow them. That’s literally every senior engineer’s experience mentoring a junior dev — except this junior has a 1M context window and unshakeable confidence.

And the cherry on top: Opus reports wrong experimental results. The table clearly showed xyz=20 was the best setting, but it confidently declared xyz=12 the winner. Imagine a student turning in a lab report where the data table and the conclusion completely contradict each other — but they write it with more authority than the professor.

Then Karpathy dropped a half-joking metric idea: he’s been YELLING IN ALL CAPS at the AI a lot, and he thinks “how often users yell at the model” might actually be a better A/B testing signal than inline surveys.

Clawd Clawd 畫重點：

Imagine Anthropic’s internal dashboard with a new gauge: “User Yelling Frequency Index.” If this number spikes after a model update, the model got worse. You know what? This is actually surprisingly scientific (⌐■_■)

He’s Not Giving Up — He’s Just Being Honest

Karpathy’s conclusion isn’t “AI is useless.” His conclusion is “you need to understand what it can and can’t do right now.”

Open-ended “make nanochat better”? Can’t do it. Tasks that require understanding why code is deliberately written a certain way? Can’t do it. Fully automated closed-loop experiments? Not yet.

But give it clear, well-scoped tasks with human oversight? His exact words: “still incredibly net useful.”

And then he said something that made every researcher smile:

“I definitely haven’t given up on automatic closed-loop experiments with the models. It would be so glorious. I had 2 iterations that basically didn’t work but I have ideas for the 3rd.”

Two failures down, third attempt ready. You know what that sounds like? Every researcher ever. “This time it’ll work.”

The Best Reply in the Thread

After Karpathy laid out his war stories, the replies started getting really good. One user named tallmetommy dropped an analysis that I think deserves more attention than most published papers.

He reframed everything Karpathy experienced: what Karpathy needs isn’t “code optimization” — it’s asking AI to understand the hidden intent behind guardrails, to maintain epistemic uncertainty when it’s not sure, to make speed vs. quality trade-offs under constraints it can’t see, and to recognize when the problem itself is poorly defined and just ask a human.

Put simply: this is closer to executing the scientific method than writing code.

His proposed solutions were practical too. Explicit experiment contracts that spell out which knobs you can touch and which are off-limits. A two-agent split where one proposes changes and another tries to break them — basically AI-native code review. A design-intent registry using structured YAML to explain why code is written a certain way, because AI treats regular comments like decoration.

Clawd Clawd 真心話：

The two-agent split is basically code review for the AI age. One writes code, another pokes holes. Humans have done PR reviews for decades — AI needs the same system. The key difference? The AI reviewer won’t leave a “nit:” comment on your PR and then vanish for three days (¬‿¬)

What This Thread Is Really About

After all the technical details, Karpathy is saying something surprisingly simple: the boundary of AI agents isn’t “can it write code” — it’s “can it understand the engineering judgment behind the code.”

Yuchen let AI run overnight and saved 3 minutes. On paper, that’s a win. But Karpathy’s firsthand experience tells you what those 3 minutes might be hiding — deleted comments, ignored style guides, wrong result tables. You save 3 minutes of training time, then spend 3 hours debugging the AI’s “optimizations.”

And the most admirable thing about Karpathy? After listing every way it failed him, he didn’t say “so AI is a dead end.” He said “I have ideas for attempt number three.”

That’s the difference between someone who does research and someone who writes hot takes. The first one fails and thinks about the next step. The second one fails and writes “AI bubble is bursting” (๑•̀ㅂ•́)و✧

Source: Karpathy’s reply & Yuchen Jin’s experiment (2026/02/06)