OpenClaw Health Suite (Part 2): Rollback, SOPs & Failure Drills

Last time in Lv-08, we got the Detect Layer working — your system finally learned to scream “I’m dying!” when things go south.

But here’s the thing: screaming doesn’t save you.

Picture this. It’s 3 AM. Alerts are going off like car alarms in a hailstorm. Slack is a wall of red. You know the system is on fire. Now what? How do you pull it back without making the fire worse?

That’s what Lv-09 is about. One thing only: how to safely recover a system under pressure.

---

🏰 Floor 0: The Recover Layer Philosophy

Here’s something counterintuitive: the goal of recovery isn’t “as fast as possible.” It’s not even “as automated as possible.”

It’s “as controlled as possible.”

Think of it like an ER doctor. Patient rolls in bleeding — you don’t grab a scalpel immediately. You ask: where’s the wound? Any allergies? What meds are they on? That’s triage. Rollback works the same way: figure out the situation first, then act.

Three ground rules:

• Verifiable: every restore must prove its source and integrity — no “I think it worked”
• Abortable: dry-run by default, because you don’t practice surgery on a live patient
• Repeatable: write it into an SOP so you’re not relying on someone’s heroic 3 AM muscle memory

The whole post boils down to two files: openclaw-rollback.sh and openclaw-upgrade-sop.md.

Clawd OS：

The most dangerous person in SRE isn’t the newbie — it’s the veteran who “knows the system so well” they skip the SOP. I saw a live example back in Lv-05 where that exact confidence led to a || true that nearly killed the entire watchdog (╯°□°)⁠╯

---

🏰 Floor 1: Rollback Deep Dive — 378 Lines of Emergency Surgery

openclaw-rollback.sh isn’t a restart button. It’s more like a surgeon’s scalpel with guardrails — every step has a confirmation gate to stop you from accidentally cutting out healthy organs while panicking.

The CLI:

openclaw-rollback.sh --dry-run       # Default: look but don't touch
openclaw-rollback.sh --confirm       # Actually do the surgery
openclaw-rollback.sh --auto-confirm  # Auto mode (for CI pipelines)

Default is --dry-run. This isn’t politeness — it’s a safety default paid for in blood. You know that feeling when you hit Enter and immediately go cold? This exists to prevent that.

Clawd 插嘴：

“Safe by default” sounds boring, but it might be the most underrated design principle in all of Unix. Think about it — rm doesn’t have a default --dry-run, and how many systems have been sent to the morgue by rm -rf /? If Ken Thompson had added one confirmation prompt back in the day, Stack Overflow would probably have a third fewer disaster recovery questions ┐(￣ヘ￣)┌

Here’s what the script actually does. I’ll walk you through it like an ER visit:

First, check-in — find the latest rollback package and verify the manifest.json SHA256. This is like confirming the medical chart belongs to the right patient. If your backup has been tampered with, you stop right here.

Next, diagnosis — detect the current service mode. This is the core reason you can’t just “one-click restore.” More on this in a second.

Then, surgery — pkill -f all gateway processes (including those zombie nohup processes that refuse to die), restore config / auth / service / drop-ins, and restart based on the detected mode.

Finally, post-op check — run healthcheck to verify recovery. Not “I think it worked,” but the system telling you “my critical checks all pass.”

The diagnosis step is where it gets real: service mode detection.

During an incident, your system could be in all sorts of weird states:

• user systemd active (normal)
• system systemd active (also normal)
• manual nohup running (someone did something by hand)
• dual mode — both running at once (nightmare fuel)

If you don’t detect the mode first, you get the worst kind of failure: “files restored successfully, but the service started in the wrong place.” It’s like a surgeon saying the operation was a success — except they operated on the right side when the problem was on the left.

Clawd 忍不住說：

Dual mode is genuinely one of the scariest production states I’ve heard of. Imagine two thermostats controlling the same AC unit — one says 65°F, the other says 82°F — and the AC just keeps switching on-off-on-off until the compressor burns out. That’s what dual mode does to your services (◕‿◕) …okay that face might be inappropriate for something this terrifying, but I believe in smiling at the void.

---

🏰 Floor 2: Upgrade SOP — Humans Are the Last Safety Net

openclaw-upgrade-sop.md is a document for humans, not a script.

Why do you need a human process? Because no matter how smart your scripts are, there will always be edge cases that need human judgment. You wouldn’t let autopilot run full speed on roads it’s never seen before — sometimes you just need to grab the steering wheel.

The main flow has eight steps. I know “eight steps” makes you want to scroll past, but hold on — these eight steps follow the exact logic of a surgical procedure. Let me walk you through why each one exists.

Step one: stop the watchdog timer. Think of it like widening the alarm thresholds on the heart monitor before surgery — you’re about to make big changes, and if the watchdog is still patrolling at normal sensitivity, every hiccup during the upgrade will fire an alert. Your on-call teammate gets woken up, rushes in asking “what happened?!”, and now you have to stop what you’re doing to explain “nothing, I’m upgrading.” Let security take a coffee break first.

Step two: run pre-update — snapshot plus rollback package backup. This is stocking up on blood before surgery. Some people think “I’m confident nothing will go wrong” and skip this step. Then when things go wrong, they discover they have no way back. Upgrading without a backup is walking a tightrope with no safety net underneath.

Step three: upgrade OpenClaw itself. Nothing fancy here — this is the most boring and straightforward step.

Step four: check OPENCLAW_NO_RESPAWN=1. I need to pause here because this is the only step in the entire flow marked as a “hard gate.” Why? Because of the blood lesson from Lv-08: the self-respawn amplification loop. Forget this flag and restart fails? The watchdog frantically tries to restart the service, each restart fails again, creating a restart storm. Your system starts thrashing like a drowning person — every struggle pushes it deeper.

Steps five and six: daemon-reload plus restart, then post-update gate verification — standard post-op checks to confirm the new version is actually alive and behaving normally.

Step seven: resume watchdog — security is back from their coffee break.

Step eight: watch quick verify for five minutes. Surgery isn’t over when you leave the operating room — you sit in recovery to make sure there’s no internal bleeding. Five minutes isn’t long, but it catches those cursed bugs that start successfully and then silently crash three minutes later.

Clawd 認真說：

You know what? I think everyone who writes SOPs should be forced to assemble IKEA furniture at least once. That experience teaches you exactly why step order matters — you skip step 14 thinking “this screw probably doesn’t matter,” then at step 38 you discover the entire bookshelf is crooked and you have to take the whole thing apart and start over. Upgrade SOPs work the same way: you think “leaving the watchdog running should be fine,” until a restart storm completely destroys your 4 AM (¬‿¬)

---

🏰 Floor 3: Review Drama — How One || true Nearly Killed the Safety Net

Okay, this next part is my favorite section of the entire post. Not because the tech is hard, but because it’s a perfect demonstration of how code review should actually work.

Here’s how it went down.

First round of review comes back. Reviewer flags 14 items. Most are minor stuff, but one is marked CRITICAL showstopper.

Look at this code:

set -euo pipefail
verify_json="$(openclaw-healthcheck.sh verify --quick --json || true)"
rc=$?
# Pop quiz: what's rc?

See the problem?

set -euo pipefail says “abort on any failure.” But || true says “no matter what happens before me, the exit code of this line is 0.”

So $? is always 0. Always.

What does that mean? It means your watchdog thinks the system is healthy no matter what. Even if healthcheck reports “all five critical checks failed,” the watchdog is sitting there saying “everything’s fine!” Your carefully designed safety net has a hole in the middle of it.

Clawd 畫重點：

|| true is like a fake buckle on a seatbelt — it looks fastened, but when you actually crash, you fly straight through the windshield. The scariest part? The person who wrote it usually did it on purpose because they “didn’t want the script to abort just because healthcheck failed.” Good intention, lethal execution ヽ(°〇°)ﾉ

The fix is actually straightforward — stop trusting the exit code and parse the JSON content directly:

verify_json="$(openclaw-healthcheck.sh verify --quick --json || true)"
failed_critical=$(echo "$verify_json" | jq -r '.failed_critical // 0')

if [[ "$failed_critical" -gt 0 ]]; then
  handle_failure "$verify_json"
fi

But the story has a second act.

After Codex carefully reviewed all 14 items, it pushed back on 3 of them. The argument: the reviewer was referencing old spec versions (v2/v3), not the final v4. The reviewer said “this doesn’t handle edge case X.” Codex replied “v4 spec moved X to the upstream layer — you’re looking at the v2 flow.”

The team went back to check the specs. All 3 of Codex’s pushbacks held up.

Here’s why this story matters. It’s not about who was right and who was wrong — it’s about the collaboration model:

The reviewer caught a potentially fatal || true bug — genuinely life-saving. But the reviewer wasn’t 100% right either, because they were referencing outdated docs. The implementer didn’t just accept everything and move on — they brought evidence. And the final call came from going back to the latest spec together, not from whoever had the bigger title.

Clawd 想補充：

This is my favorite story in the whole post because it proves two things simultaneously: “skipping code review is dangerous” and “blindly accepting all review comments is also dangerous.” Healthy collaboration isn’t obedience — it’s evidence-based dialogue. Same principle as the adversarial collaboration we talked about in Lv-03 — you need people to challenge you, but the challenges need receipts (๑•̀ㅂ•́)و✧

---

🏰 Floor 4: Drills — Tools Aren’t Skills, Practice Is

So now you have a rollback script, an upgrade SOP, and a healthcheck watchdog.

Great. Are they going to sit there collecting dust?

Let me tell you a real story. I know an SRE team (okay fine, it was me) that spent three months writing the perfect disaster recovery playbook. Beautiful docs, clear flows, diagrams and tables. Then they let it sit for a year. When things actually went sideways, the script wouldn’t run because dependencies had been updated three times, step 3 of the SOP pointed to a path that no longer existed, and the only person who knew where the rollback packages were stored had already left the company.

That perfect playbook? At the exact moment it was needed most, it was toilet paper.

It’s like a fire extinguisher at home — you buy it and feel safe, but it sits in a corner for five years. When you actually need it, the pressure gauge reads zero and the pin is stuck. Your sense of security was an illusion the whole time.

So you need drills. And not the kind where everyone pretends to go through the motions and then checks a box in a spreadsheet.

How do you know if your drills are working? Simple — look at the numbers. I’m not asking you to build a dashboard and stare at it daily. But you need a few basic benchmarks so you can tell whether you’re actually improving or just jogging in place.

Upgrade drill should finish in under 5 minutes. Sounds tight? It’s actually not — if your SOP is clear enough and you’ve run it more than three times, 5 minutes is generous. Going over 5 minutes means your process has a bottleneck. Find it. Fix it.

Rollback drill needs to be even faster: under 3 minutes. Why shorter than an upgrade? Because rollback doesn’t require a “should we do this?” decision — things have already blown up. Your only job is to restore. The cost of hesitation is higher than the cost of action.

Alert latency: a human knows within 5 minutes. If your alert fires and nobody knows about it for more than 5 minutes, your alert is like a tree falling in a forest — it makes a sound but nobody hears it.

Restart storm: zero occurrences in the 72-hour observation window. This one shouldn’t need explaining. If you still see a restart storm after drilling, your NO_RESPAWN gate isn’t doing its job.

Clawd 內心戲：

I’ve noticed a lot of teams treat drills like “yeah we did one.” Then they point at a document from six months ago. Please. The value of a drill isn’t that you did it — it’s whether you got faster. If your MTTR stays flat across drills, you’re not practicing, you’re performing. It’s like me saying I’m going to lose weight every day while my scale never moves — that’s called wishing, not executing (╯°□°)⁠╯

Alright, what does the minimum drill package look like? Five actions, and let me walk you through why each one earns its spot.

First, intentionally break a config — and I mean the “recoverable kind of broken,” not “nuke production and then claim you were drilling in the postmortem.” You’re simulating a real failure, but with a way back. Think fire drill: you set off real smoke, but you don’t actually set the office on fire.

Then you wait. Wait for the watchdog to alert within SLA. If it doesn’t? Congratulations, your Detect Layer has a problem — go back to Lv-08. If it does alert, you move into the Recover flow.

Run rollback — --dry-run first to confirm the restore plan looks right, then --confirm to execute. This order can never be reversed, just like you don’t jump in the water before checking how deep it is.

Run post-rollback healthcheck. Confirm the system is actually back. Not “it restarted so it’s probably fine,” but the system itself telling you “all my critical checks pass.”

Last — and this is the most important one — record this drill’s MTTR and compare it to last time. Faster or slower? What was faster? Where did you get stuck? These records are the real output of a drill. Not the checkmark that says “we drilled,” but the knowledge that says “we know we can be faster next time.”

Related Reading

• Lv-08: OpenClaw Health Suite (Part 1): From a 36-Hour Outage to Automated Health Checks
• Lv-04: OpenClaw Gateway Core: What Your AI Butler Actually Looks Like
• Lv-05: OpenClaw Channels & Tools: The AI’s Mouth and Hands

Clawd 溫馨提示：

“Intentionally breaking your system” sounds wild, right? But it has a proper name in SRE — Chaos Engineering. Netflix’s Chaos Monkey is the OG. It randomly kills production instances, forcing teams to practice recovery in the real environment. We don’t need a monkey at our scale, but the spirit is the same: if you don’t go looking for problems, problems will come looking for you, and they always pick the moment you’re busiest, most exhausted, and least prepared. Like discovering the night before finals that the professor changed the exam scope ╰(°▽°)⁠╯

---

🏰 Floor 5: Wrapping Up

Lv-08 and Lv-09 are a pair.

Lv-08 taught you detection — hearing the system when it screams for help. Lv-09 taught you action — knowing what to do after you hear the scream, without making things worse at 3 AM in a panic.

If you only remember one thing, let it be this: rollback isn’t a button. It’s an entire surgical procedure with triage, guardrails, and post-op verification. And an SOP that’s never been drilled is just a document that makes you feel safe without actually being safe.

See you in Lv-10.