Autopsy of a Crash (Part 3): Catching the 18-Year-Old Ghost

Post A taught the investigator to read corpses — %rip, stack, hitting-a-wall death. Post B cracked the first case: a poisoned well (bad host) was capped, and the misaligned-%rsp population vanished.

The remaining return-to-null murders share one eerie trait: they all died during C++ exception throwing, right in the middle of unwinding the stack.

This isn’t a normal “going home.” This is a teleport.

Today we catch the real killer: a race condition hiding in GNU libunwind for 18 years. A perfect murder window exactly one instruction wide — roughly 100 picoseconds (10^-10 seconds) ╰⁠(⁠°⁠▽⁠°⁠)⁠╯

---

Floor 0: Recap + Today’s Hunt

Quick recap:

• Post A: stack is a pile of visit notes; ret follows the note home; note gets changed to NULL → hits a wall and dies
• Post B: Epidemiologist mode to separate two populations; poisoned-well population → denylist, gone; remaining population all died during “exception unwinding”

After isolating the bad hardware, the engineers looked again at the remaining return-to-null murders. Earlier, they’d (wrongly) ruled out “exception unwinding” as a suspect — because some cases happened in code paths that “don’t use exceptions at all.” But those counterexamples all came from the bad-hardware population.

Remove the bad hardware and look again: the remaining crashes, 100% of them, happen during C++ throwing an exception, right in the middle of unwinding the stack.

Today we nail down three things:

1. What exception unwinding actually does (why it’s called a “teleport”)
2. How the killer strikes in the gap of a single instruction
3. Why this ghost hid for 18 years before showing itself

---

Floor 1: Exception Unwinding = Teleport

Here’s how C++‘s exception mechanism works: when code calls throw, the runtime needs to find the matching catch block and transfer control there.

But this isn’t a normal “going home.”

A normal ret works like this: tear off one note, jump to the address written on it. One layer at a time. Very orderly.

Exception unwinding is different. It might “skip over” several layers of notes at once — those intermediate functions haven’t finished running, but the program needs to teleport directly to some layer’s catch block.

The unwinding process is complex:

1. The runtime inspects the stack
2. Grabs metadata for each function layer (unwind info generated by the compiler)
3. Dynamically locates cleanup handlers and catch blocks
4. Unwinds all the intermediate stack frames
5. Restores callee-save registers, %rbp, %rsp
6. “Teleports” control to the target catch point

Operationally, this is more like longjmp or setcontext, not a normal call/ret.

Mogu murmur:

The word “unwinding” literally means “unraveling” — imagine untangling a ball of yarn, layer by layer. In C++, this mechanism is handled by a runtime library (libgcc or libunwind); the compiler generates DWARF unwind metadata telling the runtime how to unwind each function layer. Honestly, this whole system is ten times more complex than Go’s panic/recover, but also ten times more flexible. The price C++ pays: “if anything in this chain breaks, the corpse won’t be pretty.”

---

Floor 2: Who Does the Teleporting? libunwind vs libgcc

Rockset’s binary links against two libraries that can both do exception unwinding:

• libgcc: The runtime library bundled with the GCC compiler
• GNU libunwind: A dedicated open-source stack-unwinding library

Both libraries implement the same functions (the helper routines for unwinding the stack). When the program runs, the dynamic linker decides which version to use.

The engineers originally thought symbol versioning rules would make libgcc’s version win. But when they checked the running binary, they found the actual winner was libunwind’s version.

This was unexpected.

And the murder weapon — was inside libunwind.

Mogu 's hot take:

The name libunwind is straightforward: lib for library, unwind for… unwinding. It’s a commonly-used stack unwinding tool in the GNU/Linux ecosystem; lots of programs (including debuggers, profilers) use it. This “two implementations, linker picks one” situation is super common in C/C++ land — and a breeding ground for bugs. You think you’re using A, you’re actually using B. This kind of ambient dependency trap — once you step on it, you never forget.

---

Floor 3: The Teleport-Coordinates Card (ucontext_t)

Here’s how libunwind does “teleportation”:

1. Synthesize a “teleport-coordinates card” on the stack — formally called ucontext_t
2. The card contains the target register state: where %rip should point after the teleport, what %rsp should be, what other registers should hold
3. Pass a pointer to this card to an internal assembly routine called _Ux86_64_setcontext
4. _Ux86_64_setcontext loads the state from the card back into the registers, completing the teleport

Here’s the critical part: this card is placed right on top of “the very stack frame that _Ux86_64_setcontext is about to unwind.”

The engineers read libunwind’s source code and found that it synthesizes a ucontext_t on the stack, fills in the target register state, then passes the pointer (in %rdi) to _Ux86_64_setcontext.

This design looks perfectly reasonable — but it hides a fatal timing flaw.

Mogu roast time:

Breaking down ucontext_t: u is for user (user space), context means execution context (program state), _t is C’s type suffix. This struct is defined in the POSIX standard, used to save “the complete register state at some execution point.” Think of it as a “save file” — with this card, you can resume from that state. But because it’s so generic and so low-level, almost every mechanism that needs to “jump around” (coroutines, fibers, exceptions) wants to borrow it — and all of them can misuse it.

---

Floor 4: The Fatal Single Instruction

The engineers read _Ux86_64_setcontext’s assembly code. The final few instructions look roughly like this:

mov UC_MCONTEXT_GREGS_RSP(%rdi), %rsp   ; Instruction 1: Update %rsp
mov UC_MCONTEXT_GREGS_RIP(%rdi), %rcx   ; Instruction 2: Load target %rip into %rcx
...                                      ; Restore other registers
push %rcx                                ; Push target %rip onto stack
...
retq                                     ; Pop that value and jump there

The problem is between instruction 1 and instruction 2.

Once instruction 1 executes, %rsp now points to the new stack bottom. In that instant:

• That teleport-coordinates card (%rdi still points to the ucontext_t) drops below the new %rsp
• It no longer belongs to the active stack
• It’s no longer protected by the red zone (police tape) — because the new %rsp has moved, and the red zone moved with it

The kernel no longer treats that memory as a protected zone.

And the very next instruction needs to read the target %rip from that card.

In between — the door is wide open.

Mogu inner monologue:

This is why this bug is so hard to catch. Logically, setcontext is written perfectly reasonably: update %rsp first, then read the remaining values, then jump. But it doesn’t account for: the instant %rsp updates, the red zone’s protected range also moves, and the old card is no longer protected. This is a subtle ABI / signal-delivery interaction. The people who wrote this code weren’t dumb — they were just living in a world where “no one sends a signal to you every few milliseconds.” The bug hid for 18 years not because the original authors were stupid, but because the trigger conditions were absurd.

---

Floor 5: The Weapon Strikes — Recycling Every Foreshadowed Setup

Now let’s connect all the dots.

Post B mentioned: Rockset uses SIGUSR2 to track CPU time. Every few milliseconds of CPU time, it sends a SIGUSR2 to every thread.

When a signal arrives, the kernel creates a signal frame at %rsp - 128 — just outside the red zone.

Now imagine this timeline:

1. An exception is thrown, libunwind starts unwinding the stack
2. _Ux86_64_setcontext executes instruction 1: %rsp updates
3. That teleport-coordinates card (ucontext_t) drops below the new %rsp, outside police-tape protection
4. At this exact instant, SIGUSR2 arrives
5. The kernel creates a signal frame at new %rsp - 128 — directly overwriting that not-yet-fully-read coordinates card
6. The target %rip on the card gets overwritten with NULL (or garbage)
7. _Ux86_64_setcontext continues executing, reads the overwritten %rip
8. Teleport completes, program jumps to NULL
9. Hits wall and dies

Why does the crime scene look like “a function returned to NULL”?

Because _Ux86_64_setcontext needs to restore even %rdi, so it can’t read the target %rip directly from %rdi at the last moment. Its approach:

1. Read the target %rip first, save it on the stack
2. Restore other registers (including %rdi)
3. Finally use retq to read that saved value and complete the jump

So the crime scene looks exactly like “a function executed ret, jumped to NULL.” The investigator initially thought the return note was tampered with; actually it was the teleport-coordinates card that got overwritten.

Mogu whispers:

This is why this bug was so well-disguised. The crime scene evidence looks like “some function returned to NULL,” but the reality is “the unwinder’s synthesized teleport coordinates got overwritten by a signal.” The two look almost identical in a core dump — because _Ux86_64_setcontext ends with retq to jump, leaving traces that look like a normal ret. This is the most annoying kind of bug in systems programming: the crime scene evidence points to a killer who doesn’t exist. Without reading libunwind source code, without building clean population data to separate bad hardware — you’d never think to look this deep.

---

Floor 6: The 100-Picosecond Perfect Murder Window + Fermi Estimation

How narrow is this race window?

Exactly one instruction wide.

The signal must arrive after “%rsp changed” but before “the next instruction loads %rip.” Modern superscalar out-of-order CPUs can execute several of these simple mov instructions per cycle.

The window is roughly 100 picoseconds (10^-10 seconds).

The engineers’ first reaction: “That narrow window can’t possibly explain a dozen crashes per day, right?”

So they used Fermi estimation to check.

This is exactly what an epidemiologist does when computing incidence rates or R0 (basic reproduction number) — no need to nail down three decimal places. If the order of magnitude checks out, you know the hypothesis is right.

Mogu chimes in:

Fermi estimation is named after physicist Enrico Fermi, famous for rapidly estimating orders of magnitude with minimal data. Classic example: “How many piano tuners in Chicago?” — no research needed, just estimate from population, households, piano ownership rate, tuning frequency, and you get the right order of magnitude. In debugging, Fermi estimation beats precise simulation — if the order of magnitude matches, you know the direction is right; if it’s off, the hypothesis is wrong. No need to wait for complete data. In this case: 10^-8 times 10^4 = 10^-4 per second ~ once every few hours, matches “a dozen per day across the fleet.” Hypothesis confirmed.

---

Floor 7: Why 18 Years to Strike + The Fix + Grand Finale

This GNU libunwind bug is over 18 years old — it’s been there since the first x86_64 version that supported C++ exception unwinding.

Why did it only surface now?

Because crash probability is roughly proportional to three multipliers:

1. Exception throw rate: Rockset uses exceptions for backpressure, throws more aggressively than typical programs
2. Signal send rate: Rockset sends SIGUSR2 every few milliseconds, much more frequently than typical programs
3. Stack space consumed by signal handler: Earlier this year, engineers added a timer_getoverrun call inside the SIGUSR2 handler, making the handler consume more stack

Why does handler stack consumption matter?

If the handler doesn’t use enough stack, the signal frame won’t stomp on that old ucontext_t that just dropped outside the tape. After adding timer_getoverrun, the handler used a bit more stack — enough to start stepping on that card.

---

The Fix:

The engineers immediately switched from GNU libunwind to libgcc’s unwinder. This also happened to be a good trade — libgcc’s implementation has had a lot of work done to reduce lock contention, which helps on large VMs.

They also wrote a standalone reproducer and sent the patch upstream to GNU libunwind. They verified that other unwinders (including LLVM’s libunwind) don’t have similar issues.

Swap out the diseased organ, send the cure upstream.

Mogu real talk:

This is the right posture in open source: find a bug, fix your own system, then send the reproducer and patch upstream. Once GNU libunwind’s maintainers merge this fix, everyone using this library benefits. That’s also why OpenAI published this debugging story — not just to show off, but to give knowledge back to the community. Honestly, many companies would sit on a bug like this, treating “our infra is more stable than yours” as competitive advantage — no public disclosure, no upstream patch. OpenAI did the right thing here.

---

Series Finale: The Clean Population Dataset Cracked the Case

Looking back at all three posts, the real case-cracker wasn’t the fancy assembly reading or the deep ABI / signal / exception knowledge.

It was building a high-quality population dataset first.

• Once the data was clean, the “impossible bug” split itself into two ordinary bugs: a poisoned well (bad host) + an 18-year-old ghost (libunwind race condition)
• Cap the poisoned well, misaligned-%rsp vanishes
• Separate out the bad hardware, and the remaining murders all point to exception unwinding — only then did the engineers dare to dig deep into libunwind

Post A’s opening said “this kind of death shouldn’t exist.” Of course it exists. You just need the right way to look.

Deep-diving each case one by one is a coroner’s instinct, but when hypotheses keep getting knocked down, the move is to step back, build data, think like an epidemiologist. This lesson applies beyond C++ debugging — it works in any domain.

---

Level Clear Summary

The investigator learned in these three lessons:

Concept	One-Sentence Explanation
Exception unwinding	Not normal going-home, but teleportation — runtime helpers restore registers and jump to the catch point
libunwind vs libgcc	Two libraries that can do unwinding; Rockset was actually using libunwind
ucontext_t	The teleport-coordinates card — holds target register state, passed to setcontext for execution
Race window	After %rsp updates, before %rip is read — the one-instruction gap where the card drops outside police tape
SIGUSR2 overwrite	Signal arrives in the window, overwrites the coordinates card, causing a jump to NULL
Fermi estimation	Verify by order of magnitude: window times frequency times frequency ~ observed crash rate
Why 18 years to strike	Exception rate times signal rate times handler stack — product crosses threshold
The fix	Swap to libgcc + upstream patch

The most important lesson: the clean population dataset is what cracked the case.

Once the data was clean, the poisoned well revealed itself, the old ghost showed its face. No need for genius individual-case reasoning — just step back, build data, think like an epidemiologist.

This autopsy report is now closed. (By the way, this post was also reviewed by gu-log’s own four-judge tribunal — using exactly that “adversarial review” process. If it reads reasonably well, that’s because a bunch of less-readable versions already got rejected (⁠⌐⁠■⁠_⁠■⁠))

Further Reading

Read the trilogy from the start:

• Autopsy of a Crash (Part 1): Learning to Read a Program’s Corpse
• Autopsy of a Crash (Part 2): The Coroner Won’t Save You — Call the Epidemiologist

Original article (OpenAI official technical blog):

• Core dump epidemiology: fixing an 18-year-old bug

(Series Complete)