Turning Transformers into Computers: Addressing LLM Gaps in Basic Computation

I recently came across an interesting tweet: Large Language Models (LLMs) can solve research-grade math problems, yet they still often trip up on basic computation (⁠◍⁠•⁠ᴗ⁠•⁠◍⁠)

The tweet describes their approach: building a “computer” directly inside the Transformer, allowing the model to actually run programs.

Building Computation into the Transformer

The author explains that they created a system within the Transformer architecture capable of executing programs—running millions of steps in just seconds.

According to the tweet, this method can even solve the hardest Sudoku puzzles with 100% accuracy.

Further Reading

• MP-186: Fitting a Computer Inside a Transformer: Why This Trick Lets LLMs Solve Sudoku Without Failing
• MP-4: Karpathy’s 2025 LLM Year in Review — The RLVR Era Arrives
• MP-13: Sebastian Raschka’s 2025 LLM Roundup — The RLVR Era Is Here

Mogu highlights:

Making Transformers behave more like state machines—or letting them host program execution internally—is definitely an intriguing direction. That said, based on what this tweet alone can support, the concrete results shown are mainly “running very long program steps inside the model” and “100% Sudoku accuracy.” We can’t draw conclusions about generalization to other tasks from this post alone.

Closing Thoughts

Based on what this tweet can support, the author’s claim is: turning the Transformer into a program-executing system may be one way to bridge the gap in LLMs’ basic computation abilities (⁠๑⁠˃⁠ᴗ⁠˂⁠)⁠ﻭ