Autopsy of a Crash (Part 2): The Coroner Won't Save You — Call the Epidemiologist

Last lesson, the investigator learned to read a corpse — figured out %rip, stack, call/ret, and how hitting-a-wall death works. But staring at a single corpse won’t reveal the killer.

This lesson has two objectives:

1. Enter the crime scene: What that sealed autopsy report (the core dump) actually is, and why certain evidence survives after death
2. Identity upgrade: From coroner to epidemiologist — use a year’s worth of citywide death records to crack the first bug

This is the middle act of the trilogy. By the end of this lesson, the investigator will catch the first ghost. The remaining victims — those who died in the middle of “teleportation” — are the main course for Post C (⁠◕⁠‿⁠◕⁠)

---

🏰 Floor 0: Recap + Today’s Mission

Quick recap of Post A:

• Program crash = casualty / sudden death
• stack = a pile of visit notes; every function call pushes a new note saying “when done, return here”
• call / ret = leave a note before going out / tear off the note and follow it home
• segfault = note gets changed to an invalid address → ret follows it → hits a wall and dies
• C++ is a city with no guardrails — anyone can tamper with anyone else’s notes

OpenAI’s batch of deaths showed this: the victims’ return notes had been changed to NULL (an empty address), or their %rsp was mysteriously shifted by 8 bytes. The investigator knows “how they died,” but not “who did it.”

Staring hard at one corpse won’t reveal the killer. Today’s mission: enter the crime scene, learn new investigation tools, and assume a new identity for this case.

---

🏰 Floor 1: core dump = The Sealed Crime Scene

When a program crashes, the operating system can save “the program’s entire state at the moment of death” into a file. This file is called a core dump.

Rockset engineers used folly’s (Facebook’s C++ library) fatal signal handler to handle crashes: the moment a crash happens, it automatically logs the stack trace (the call chain) and uploads the core dump to Azure blob storage for safekeeping.

Rockset’s query nodes are replicated, so a single crash doesn’t hurt users much. But every crash represents a bug that needs fixing.

Mogu twists the knife:

The name core dump comes from the ancient days — back when magnetic-core memory was the standard, so “dumping memory” was literally called core dump. Modern computers haven’t used magnetic cores for decades, but the name stuck. When engineers say “grab that core for me,” they’re talking about this sealed-crime-scene file.

---

🏰 Floor 2: The Who-Called-Whom Chain — Tracing with %rbp

Post A mentioned a detail: Rockset compiled with the -fno-omit-frame-pointer flag. Now let’s explain why this matters.

Inside every stack frame (visit note), besides the return address, there’s also the “previous layer’s %rbp.” This lets the investigator start from the topmost note and trace upward along the %rbp chain to see the complete call sequence: who called whom, who called whom before that.

Some compilers omit %rbp for performance (with the -fomit-frame-pointer flag) — this saves one register per function, slightly faster. But the cost: when a crash happens, investigators can barely reconstruct the call chain.

Rockset did the opposite with -fno-omit-frame-pointer — originally probably just for debugging convenience. Turns out this flag saved the engineers’ lives in this investigation. Without this visit-note chain, investigators couldn’t even piece together “who the victim met before dying.”

Mogu murmur:

Post A’s foreshadowing about “a compiler flag saving their lives” is this. -fno-omit-frame-pointer looks like just a debugging convenience option, but in a scenario where “notes get tampered with and the entire stack might be corrupted,” this visit-note chain is the investigator’s only tool for reconstructing what happened before the crime. This is why many production systems deliberately enable this flag — sacrificing a tiny bit of performance for observability.

---

🏰 Floor 3: The Police Tape Line (red zone) — Why Evidence Survives After Death

Now for a crucial detail: why can investigators see the contents of notes from “functions that already returned”?

Logically, after function B finishes and executes ret, its stack frame “is no longer valid” — that memory can be overwritten by the next function. But the engineers wrote in their report: “X, this just-popped stack frame, looks normal except the return address is NULL.” How did they see that?

The answer is the red zone (the police tape line).

When a crash happens, folly’s signal handler (next floor) runs on the same thread’s stack. This handler will trample old notes that already returned — but the evidence inside the police tape (the last 128 bytes) survives.

This is why the engineers could say: “X, this just-popped stack frame, looks normal except the return address is NULL.” The police tape preserved their final bits of evidence.

Mogu real talk:

The name red zone is vivid — “red zone, no entry.” In the System V ABI definition, these 128 bytes are “space leaf functions (functions that don’t call anyone else) can use for local variables without adjusting %rsp.” For investigators, the value is: the first responder on scene (signal handler) will trample the scene, but evidence inside the tape survives.

---

🏰 Floor 4: The First Responder (signal handler)

When a program crashes, the kernel sends a signal to the program. For a segfault, the kernel sends SIGSEGV (segmentation violation).

A program can register a signal handler ahead of time — when this signal arrives, don’t die immediately; first let the program run a designated piece of code. Rockset uses folly’s fatal signal handler, which records the stack trace and prepares the core dump when a crash happens.

The problem: this handler runs on the crashing thread’s own stack — sharing the same lifeline as the deceased.

Now here’s some foreshadowing: Rockset uses signals more aggressively than typical programs.

They have a mechanism called coarse_thread_cputime_clock: using the timer_create API, every few milliseconds of CPU time, a SIGUSR2 (signal user 2) gets sent to every thread, letting the handler update a thread-local CPU time value. This tracks “how much total CPU time this query has used.”

This “signal every few milliseconds” design — just file it away as background knowledge for now. In Post C, it becomes part of the murder weapon.

Mogu roast time:

In the name SIGUSR2, SIG is signal, USR2 is user 2 — “user-defined signal number 2.” Unix systems reserved two signals (SIGUSR1 and SIGUSR2) for programs to define their own purposes. Rockset uses SIGUSR2 for CPU time tracking. Every few milliseconds — this frequency is very aggressive for a typical program. Remember this setup; Post C will use it.

---

🏰 Floor 5: Coroner vs Epidemiologist

Now we reach the heart of this lesson.

OpenAI’s engineers were stuck for days. They:

• Carefully examined several core dumps
• Deep-dived into one misaligned-%rsp case — reconstructing the history before the crime, reverse-engineering from stack and register contents
• Read kernel source code, read Azure-specific kernel patches
• Ran stress tests

Tried everything. No results.

This was the turning point of the entire investigation: they decided to stop staring at individual core dumps and instead collect high-quality population data.

Why were they stuck for so long? Because they kept assuming “this is the same bug.” Some cases had return addresses of NULL, some had %rsp shifted by 8 bytes — symptoms were somewhat similar, yet not quite identical. The engineers treated them as the same bug, but every hypothesis had counterexamples, and they couldn’t catch anything.

An epidemiologist would ask: Is this really one disease? Or are there two diseases mixed together that just happened to be discovered at the same time?

Mogu PSA:

This is what I consider the most brilliant insight in OpenAI’s original article. Many hard-to-debug problems aren’t hard because of technical details — they’re hard because “you think you’re solving one problem, but actually two unrelated problems are mixed together.” The engineer’s instinct is to deep-dive a single case — coroner mode. But when hypotheses keep getting knocked down, the move is to step back, collect population data, and look for patterns. This technique works in any domain.

---

🏰 Floor 6: Building a Clean Dataset

After deciding to switch to epidemiologist mode, the first question: where does the data come from?

The engineers initially wanted to use log text searches to automatically grab all cases. But stack-corruption bugs have a catch: the logs themselves are corrupted. Because the stack got mangled, the logged stack traces are also garbled. Text searches would have false positives (grabbing cases that aren’t this bug) and also miss cases (this bug, but the log looks different).

They changed approach:

1. Had ChatGPT write a script
2. The script downloads just the header of each core file (no need to download entire GB-sized files)
3. Extracts register values
4. Filters out known false positives using logs
5. Automatically labels each crash as: return-to-null (return address is NULL) / misaligned-stack (%rsp misaligned) / other
6. Runs in parallel across an entire year of production Rockset core dumps

This is “filing every corpse in the city for a year.”

The moment clean data came out, correlations jumped out.

What they thought was one bug turned out to be two completely unrelated crash populations.

Population	Characteristics
return-to-null	Scattered across many clusters, many regions; recently increased but no clear start date; no clean infrastructure boundary
misaligned-stack	All from one region; clear start date; never happens on nodes that have been up for a while; although spanning multiple Azure VMs, pattern looks like “one physical machine is broken, hurting VMs that happen to land on it”

The two populations had completely different patterns.

The engineers had been mixing them together, so every hypothesis had counterexamples — because hypotheses about return-to-null couldn’t explain misaligned-stack cases, and of course that wouldn’t work. View them separately, and the answers become clear.

Mogu going off-topic:

“Thought it was one bug, actually two” — this lesson is critical. The engineers initially (wrongly) ruled out hardware problems because “it happened across multiple regions, multiple hardware types.” But that was because they were viewing two bugs as one. Once separated, the misaligned-stack population’s pattern was obvious: all from the same region, clear start date, only happens on freshly booted nodes. This is textbook “one physical machine is broken.”

---

🏰 Final Floor: Bug #1 — The Poisoned Well (John Snow)

In 1854, cholera broke out in London, killing hundreds within days. The prevailing theory was “miasma” — bad air causing disease.

A doctor named John Snow didn’t buy it. He drew a map, marking each death’s address, and noticed a pattern: deaths clustered around the Broad Street water pump.

John Snow went to inspect that pump and found the well water had been contaminated by a nearby cesspool. He convinced authorities to remove the pump handle so people couldn’t draw water from it anymore.

The cholera stopped.

This is the origin story of epidemiology. John Snow had no microscope, no bacterial culture. What he used was population data and pattern recognition.

The engineers added that host to the denylist (capped the well).

They stress-tested for weeks in a controlled environment and couldn’t reproduce the register-corruption phenomenon — they never figured out exactly how that CPU was broken. But it didn’t matter. After taking that machine offline, the misaligned-stack crashes disappeared.

Just like John Snow didn’t need to know what cholera bacteria looked like. He only needed to know: remove that pump handle, and the deaths stop.

The engineers also made a few improvements along the way:

• Enhanced the fatal signal handler to record register state. Now just looking at logs can detect similar issues without downloading entire core dumps
• Changed the control plane to prefer VM reuse over recycling. This makes “bad nodes” easier to catch — if VMs on the same physical machine have unusually high death rates, that’s suspicious
• Updated the runbook

A poisoned well — just cap it.

Mogu twists the knife:

The John Snow story is the classic opener for epidemiology. In 1854, the mainstream believed in “miasma”; Snow used data to prove them wrong. He had no microscope, no petri dishes — just a map and a pile of death records. This parallels OpenAI’s engineers’ situation: they couldn’t reproduce the bug in a lab, but clean data was enough. Modern medicine calls John Snow “the father of epidemiology” — and his methodology works just as well when debugging infrastructure.

---

Now that the misaligned-stack population is separated out, the remaining return-to-null group suddenly makes much more sense.

The engineers had earlier “ruled out exception unwinding (exception unwinding) as a possibility” because they thought they’d found counterexamples: some crashes happened in code paths that “don’t use exceptions at all.”

But those counterexamples — all came from the bad-hardware population.

Remove the bad-hardware population and look again: the remaining crashes all happen during C++ exception throwing, while the stack is being unwound.

What is this exactly? What’s that “teleportation” about? How does the murder weapon strike?

Post C.

---

🎓 Level Clear Summary

The investigator learned in this lesson:

Concept	One-Sentence Explanation
core dump	The sealed crime scene — a complete state snapshot at the moment of crash
Visit-note chain	A linked list using %rbp, reconstructing “who called whom”
red zone	The police tape line; the kernel promises not to cross; these 128 bytes of evidence survive after the crash
signal handler	The first responder on scene, tramples old notes outside the police tape
Coroner vs Epidemiologist	Staring at individual cases vs looking at the whole population for patterns
Two populations	return-to-null and misaligned-stack are two unrelated bugs
Poisoned well (John Snow)	Bad host → denylist → crashes disappear

The investigator also learned the most important lesson: when coroner mode gets stuck, switch to epidemiologist mode. Build clean population data, ask “is this really one bug?” — the answer is often “no, it’s two mixed together.”

The bad-hardware population is solved. The remaining return-to-null crashes all happen during “C++ throwing an exception, in the middle of unwinding the stack.”

What is that “teleportation”? How does the killer strike in the gap of a single instruction?

Post C — catching the 18-year-old ghost.

🔗 Further Reading

Post C will cover exception unwinding, the race condition in libunwind, and the 100-picosecond murder window. Want to warm up first, or like this tower-climbing style?

• Autopsy of a Crash (Part 1): Learning to Read a Program’s Corpse
• Unix Signals 101 — SIGUSR1 vs SIGTERM vs SIGKILL

(Post C coming next)