Five Tests Between an AI Demo and an Owned Workflow
A demo has not become an owned workflow until it clears five minimum tests, none of which are about the model: it starts without its author, its decision boundary is written down and enforced, its signals separate the states you act on, its work leaves evidence you can read from outside it, and its recovery has been run rather than documented.
The demo worked. Everyone in the room saw it work. Six months later, the thing it demonstrated is not part of how the business runs, and no one can say exactly when it stopped being a priority or whether it was ever really started.
That is not a story about a model falling short. The demo proved capability on one bounded path under prepared conditions. It did not prove that the business could rely on the result when the builder was absent, a dependency changed, or a bad output went unnoticed.
For five editions I have written about specific ways that gap stays open: instructions that were never enforced, an approval that was not attached to anything, a workflow nobody owned, a reboot that restored machines before their capabilities, an alarm that could not tell a clean reading from a failed one. Each one was a diagnosis. This week is the instrument.
Below are five tests. Scoring one workflow against them takes about an hour with the evidence you already have; only the recovery drill needs real calendar time behind it. None of them require you to understand the model, and each one came out of a failure that was expensive to find. If you have an AI pilot that impressed people and then quietly stalled, you can run these against it this week and end up with something better than an opinion: a specific list of missing evidence.
Why "the model works" answers the wrong question
The instinct after a good demo is to ask whether the model is capable enough. Capability is the part the demo already proved. The question that decides whether the business can depend on the workflow is different: does this keep working, and how would we know if it stopped? The five tests below are that question, split into the places it fails.
Test 1: does it start without its author?
Open the workflow and ask who or what begins it. If the honest answer is "someone opens a notebook," "someone pastes the week's data in," or "Priya runs it Monday mornings," you have a demo with a human engine.
The distinction is not whether people are involved. Plenty of good workflows require judgment, approval, or a human in the middle, and should. The distinction is whether the workflow starts on its own terms, from a scheduled time or a real business event, or whether it starts because a particular person remembered.
Pass looks like: a recent run that started from its schedule or its business event without the author touching it, and a named person accountable for the outcome that run produced.
I wrote an entire edition about this after finding that a system of mine had gone quiet for reasons nobody owned. The lesson generalized: initiatives stall not because the model is incapable, but because nobody owns the full path from trigger to action to verification to recovery. When no one owns the whole path, a dead dependency reads as a clean zero, and a clean zero looks exactly like a quiet success.
Test 2: is the decision boundary written down and enforced?
Ask what the AI is allowed to decide by itself, and ask for the answer in writing.
Most teams discover at this point that the boundary exists only as a shared understanding, which means it is not a boundary. It drifts as people get comfortable. It differs between the person who built the thing and the person who inherited it.
When a decision does need a human, the approval has to be real, and I have come to think that means four specific things: the approval writes a record, the record binds the exact content that was approved, the approval expires, and the send returns the platform's own receipt rather than the system's assurance that it sent. Drop any one of those and you have a button that makes people feel better.
Pass looks like: a written list of what the system may do unsupervised, what requires approval, and what it must never do, plus evidence from a recent run or controlled test that every applicable class is enforced. An allowed action succeeds. An approval-required action cannot execute without a fresh record bound to the exact action. A prohibited action is rejected and records why. When execution crosses a system boundary, the downstream system returns its own receipt.
Test 3: can its signals separate the states you act on?
Every AI workflow reports something: a count, a status, a dashboard tile, a weekly summary. Take one that a decision actually depends on, a decision-bearing signal, and ask two questions about it. Which decision changes depending on this value? And what would this number look like if the thing producing it had failed entirely?
Signals fail in two different directions, and you have to check both.
The first is saturation: the threshold is so wide that everything clears it. I had a tracker that reported thirteen of thirteen active items stale, because the rule flagged any item whose claim had passed its expiry, a window of a couple of minutes. The number was accurate and useless for the one decision I was asking it to inform that day, which item to pick up first, because a measure that includes the entire population ranks nothing.
The second is silence: the check stops observing and reports clean anyway. Mine ran, failed to get a valid reading, and reported the failure as a healthy result. For two nights nothing new was backed up while the status said fine.
Those are not the same defect, and they need different fixes. Saturation is a calibration failure. Silence is an observation failure: the signal has stopped watching and still answers.
Pass looks like: for each decision-bearing signal, a value it has produced, in history or under a controlled test, that sent a named decision the other way; then, in a controlled test or an approved maintenance window, fail the underlying check on purpose and confirm the signal reports a state you can tell apart from healthy.
Test 4: can you reconstruct the run and corroborate its effects?
Pick yesterday's output. Ask your team to show what the system examined and what it did, using something other than the system's own report about itself.
This sounds pedantic until the first time an AI workflow is confidently wrong and you need to reconstruct what it saw. A log the system writes about its own behavior is an assertion. Evidence from outside it, the platform's receipt, the counterparty's record, the artifact someone else received, corroborates the claim instead of repeating it.
Pass looks like: for any recent run, a durable trace of inputs, authority, actions, and output committed outside the acting component, plus independent evidence for the resulting effect: a downstream receipt, consumer acknowledgement, or immutable artifact another component received.
Test 5: has the recovery been run, or only written?
Ask what happens when it breaks, then ask when someone last did that on purpose.
There is usually a document. There is usually not a date. A recovery path that has never been executed is a plan, and plans have a poor record against reality: the credential expired, the runbook references a service that was renamed, the one person who knew the manual step has left.
This is the same shape as the first lesson this newsletter ever ran. Across the window I measured, the recurrence rate of review findings had not visibly shrunk, which was consistent with, but did not prove, the hypothesis that written instructions without enforcement are not control flow.
And restoring a machine is not restoring a capability. A system is recovered when its dependencies, its controls, and the workflow itself are proven working again, and the proof is written down.
Pass looks like: a deliberate interruption, a recovery that then ran end to end against the system as it stands now, and a dated record of both. An attempted drill that did not finish is a fail with a date on it.
Scoring it
Give the workflow one point per test. Be strict; the point of a bar is that things fail it.
Five out of five means the workflow clears this ownership screen. It does not certify complete ownership. Anything below five is not a maturity score; it is a list of missing evidence. Finding out you are at two is not bad news, it is the first useful information you have had about this project since the demo.
These tests are ordered for reading, not as a proven dependency chain. Start with the failure that makes another answer untrustworthy or creates the greatest business consequence.
These five are not exhaustive. They are the minimum bar earned by five editions of failures.
What to do with a failed test
Run this against one workflow. One, not a portfolio. Give yourself an hour, write the five tests down the side of a page, and fill in pass, fail, and one line of evidence for each. You do not need me for the first pass. If all five pass, this screen found no gap; that is not complete certification.
If the failures cluster, that pattern is the work. That is what the Readiness Scan is for: thirty minutes on one workflow, to baseline where it stands, define what good looks like and what breaks trust, map the constraints around tools, data, handoffs, ownership and timing, and choose which line of work comes next.