The Machines Came Back. The Workflow Didn't.

A reboot restores a machine, not a capability: an AI system is recovered only when its dependencies, controls, and workflows are proven working again and the proof is recorded.

One Saturday morning this month, the power died at the site that runs my AI operation. Four machines went dark at once: no shutdown sequence, no warning, just a hard cut. Seven and a half minutes later the power was back, and the first machine had already restarted itself.

By the signals attached to that machine, it had recovered. The host answered. The service runtime was up. Dashboards showed processes running. Two of my databases reported themselves healthy.

Here is what was actually true. More than half of the services on that machine were down. The two databases reporting healthy had no network connection at all. The other three machines stayed dark until a person pressed their power buttons, and the last capability was not proven back until roughly seven hours after the cut. The workflows the business actually runs on had not returned; a green dashboard had.

Last week I wrote that AI pilots stall because nobody owns the workflow. This week reality ran the test on my own systems, and it asked a sharper question: when something interrupts your AI operation, who owns getting it back, and how would they prove they had? In most organizations the honest answer to the first half is "whoever built it," and the honest answer to the second half is a shrug.

The reboot was the easy part

A reboot is a machine property. Recovery is a workflow property. The gap between them is where this incident lived, and where most AI operations are exposed.

AI workflows widen that gap. A single capability can cross a model endpoint, a retrieval store, a policy gate, a credential boundary, a queue, and the record it is supposed to leave behind. Every component can be green while the workflow is unavailable, or available without the controls that make it safe to run.

"The server is back" is a claim about hardware and an operating system. It says nothing about whether the services started, whether they can reach each other, whether an end-to-end operation succeeds, or whether that operation leaves the evidence it is supposed to leave. Each of those is a separate claim, and each one can be false while the claim below it is true.

Too many organizations running AI workflows have monitoring for the bottom claim and intuition for the rest. That is exactly backwards. The bottom claims are cheap to verify and rarely lie. The top claims are the ones your revenue depends on, and they are the ones nobody instruments.

Ten seconds, thirty-six services

The mechanism was not exotic. It was an ordering assumption nobody had ever tested.

When the machine restarted, it began restoring services ten seconds after its network layer started, before the machine's own network address existed. Everything that binds that address failed to start, and the runtime does not retry a failed restore; it leaves the service stopped, silently. Twenty-one of thirty-six services sat exited. Even restarting them was not enough: a restart brings back the process, not its wiring, and thirteen came up running with no network attached at all.

The false green went one layer deeper. Two databases reported healthy while they had no network at all, because their healthchecks probe from inside the container. The check asked "am I up?" when the question that mattered was "can anyone reach me?"

One component told the truth: the secrets store sealed itself on restart, by design, and refused to hand out credentials until a person with the right keys unlocked it. Keeping that step away from the machine's own service account limits what a compromised host can reach. Note the asymmetry: the fail-closed component was the only one that behaved exactly as its design declared.

The recovery ladder

The incident gave me a vocabulary I now apply to every workflow I operate. Recovery is a ladder, and each rung is a distinct claim that requires distinct evidence.

Powered: the hardware and operating system returned. Running: the processes started. Ready: the things each process depends on are reachable, and it can prove it. Capable: one real end-to-end operation succeeds. Receipted: that operation left a verifiable record that it ran and what it did.

My outage was a tour of the gaps between rungs. Powered without Running: twenty-one stopped services. Running without Ready: thirteen processes with no network. Ready without Capable: databases back and reachable while their clients still failed, until the clients were restarted and abandoned their dead pre-outage connections. And everywhere, green lamps at one rung being read as proof of the rungs above.

The top rung was not the incident report itself. It was the end-to-end checks that passed and the durable records they produced. The incident report bound those receipts to the timeline: what failed, what changed, which checks passed at each rung, and who verified the return. Without that binding, recovery is still a rumor.

This ladder is also a contract you can write down for any workflow that matters: a declared startup order instead of a remembered one, readiness checks that cross the seams between components instead of probing themselves, one end-to-end probe per workflow, a receipt when it passes, and a named owner for the whole climb. None of that requires new technology. It requires deciding that "it looks up" is not a rung.

What we changed, and what stayed honest

The fixes were small, which is the point. The boot ordering became a declared dependency instead of an untested assumption, and remote power-on was armed on every machine. Both are marked unproven until a real power cycle exercises them, because a recovery contract that overstates its own coverage is the false green all over again, one layer up. The full restoration sequence, in dependency order, also moved from one person's head into the written incident record.

The outage also called in a debt it did not create: one service had run for five weeks on a connection made before an old naming scheme was retired, and the forced reconnect exposed it. Interruptions do not manufacture fragility. They audit for it, on a schedule you do not choose, with an escalation path you do not control.

A UPS is not a recovery plan

The obvious response to a power cut is a battery, and one is on my list. A UPS can keep a brief power event from becoming an interruption at all, and can turn a long one into a controlled shutdown. What it cannot do, by itself, is prove that the workflow can recover to the top rung. The metric that matters is not time between interruptions. It is whether recovery is a property of the system, exercised and receipted, or a performance by whoever built it.

The operator test

You do not need an outage to find out where you stand. Three questions do it.

If power died to the systems behind your AI workflows right now, what would come back without a human who remembers things? Not what should come back. What has actually been observed to come back, in a test someone can point to.

What do your health signals actually probe? If a check runs inside the thing it is checking, it can only vouch for that thing's inside. Ask which of your checks cross a seam: from one service to its dependency, from your system to the outside world. Checks that cross seams would have caught what my outage threw. Checks that do not can remain green while the seam is broken.

When your system says it has recovered, what is the receipt? A dashboard asserting health is the system's opinion of itself. A completed end-to-end operation with a durable record is evidence. If your recovery story ends at a screenshot of green tiles, it is a story, not an audit trail.

Minutes after the power came back, my monitoring implied the system had returned. The last capability was proven back roughly seven hours later, and only because a person knew the sequence: which control to unlock first, which wiring to reattach before trusting any green lamp. That gap between the reboot and the receipt is the part your organization either owns or does not.

If recovery depends on the person who built the system remembering what to do, the organization does not own the workflow. That person does.