The Alarm That Always Fires

A signal is only useful if it separates the states a decision depends on, and it loses that power two different ways: saturation, where the threshold is so wide everything clears it, and silence, where a failed observation is reported as a healthy one.

Last week I wrote that a reboot is not a recovery. This week the same systems taught me the sharper version, and it cost me two nights of backups to learn.

A monitor reported that everything was fine. It was not reading anything at all. For two nights nothing new was copied anywhere: the most recent backup of my working data was the one taken before dawn on the night of the crash, and it kept getting older. By the time I caught it, the freshest copy I had was more than two days old. Had a disk failed in that window, I would not have lost everything, I would have lost the work done since that copy was taken and, worse, I would have been finding that out at exactly the wrong moment.

Nobody was asleep at the switch. The person accountable for that backup is me, the check was green, and green was the only answer that check was capable of giving. That is the failure worth your attention: not an operator who missed an alarm, but an alarm that could not have sounded.

Here is the shape of it, and it is not a technology problem. A signal is only useful if it separates the states your decision depends on. There are two ways it stops doing that, and they look nothing alike.

Saturation and silence

A signal can fire on everything, or it can keep reporting clean after it has stopped observing. Most operations have both, and both destroy the same thing.

The first is saturation. My own work tracker reported that thirteen of thirteen active items were stale. Every single one. The rule flagged any item whose claim had passed its expiry time, a window of a couple of minutes, which meant it counted an item picked up four minutes ago the same as one abandoned for days. On the day I looked, every active item cleared that bar. The number was not lying. It was just useless: a count that includes the whole population ranks nothing, so it cannot tell me which item to pick up first, which is the only question I was asking it. And counts like it still get read out in meetings.

The second is silence. A check reports zero, and the zero is not an observation at all. Mine ran, failed to produce a valid reading, and reported that failure as a clean result.

These are not the same defect, and it matters that you can tell them apart. Saturation is a calibration failure: the signal is still watching the world, but its bar is set so low that everything clears it. Silence is worse, because the signal has stopped watching and reports anyway. One says everything is urgent. The other says nothing is wrong. Neither one lets you separate the state you are in from the state you fear, and that separation is the entire reason the number exists.

What it cost me

A crash zeroed a set of files in the repository holding my operator state. In the hours after, both protection paths stopped refreshing, for unrelated reasons: the nightly encrypted backup hit a lock left behind by the last run that never exited cleanly, and the daily snapshot was refused by a secret scanner.

For two nights, nothing new was backed up. The health check reported status OK, zero changes, ready to snapshot, the entire time.

It was not lying about what it saw. It never saw anything. The check asks the storage system what has changed across the whole repository. That question failed outright, because the damaged files made it unanswerable. But the code that asked it treated a failed question and an empty answer as the same thing, and an empty answer means nothing has changed. So a repository whose overall state could not be read reported as spotless.

That fold is the whole concealment. Failure became emptiness, emptiness became zero, zero became fine.

I found it by accident. I happened to ask the same question about one folder instead of the whole system, that narrower question still worked, and it came back with four changed files against the monitor's zero.

Everything failed toward reassurance

That is the part worth taking to your own operation.

The snapshot step logged that changes were not captured. The next step still printed the word verified, because it was comparing my copy against the backup copy and they matched, since nothing new had been written. Then the scheduler noticed the job kept failing and disabled it, which removed even the failing signal.

Three components, three different behaviors, all resolving to quiet. Nothing in that chain was designed to deceive. Each piece degraded in the direction that produced the least noise, and the sum of those small mercies was a system that had stopped protecting anything and said so nowhere.

Silence is the default output of a broken system, because a system that has stopped working has also stopped complaining.

What I would not have caught

I want to be precise about my own limits here, because this is where the lesson gets uncomfortable.

In the middle of this, I needed to confirm that a security check still worked. So I fed it something that should obviously have tripped it, and it came back clean.

The string I chose was the vendor's own published example credential, the one that appears throughout their documentation and that scanners deliberately ignore for exactly that reason. My test used input the detector is built to disregard. A clean result from it proved nothing whatsoever, and had I stopped there I would have recorded a passing check that had never checked anything.

I re-ran it with something the detector had no reason to ignore, and it fired correctly. That is the rule worth keeping: after you change any check, prove it still catches a real violation before you trust its next green.

Note what happened, though. I had written a test that could not fail, while fixing a monitor that could not fail, while writing about numbers that separate nothing.

That is the honest shape of this problem. It is not a competence failure, it is a default. Systems and the people building them both drift toward the arrangement that produces fewer alarms, and you do not notice, because not noticing is exactly the symptom.

What a real measurement requires

Three properties, and none of them require new tooling.

It must separate the cases you act on. Before you accept a number, ask which case it would have reported differently, and whether that difference is one you could do anything with. My tracker could have come out differently, an item claimed inside its window would not have flagged, and that is exactly the point: the bar was set so low that on any ordinary day everything clears it, so the count arrives full and ranks nothing.

It must distinguish failure from emptiness. Zero findings and could-not-check are different states, and code that folds them together is the specific bug I shipped. The fix is not vigilance, it is a value that cannot be mistaken for health: make the probe raise, or return something no one would read as good news.

It must alarm on absence, not just on errors. Every error-based alert assumes the checker is alive to send it. One dumb control would have broken the silence: no successful backup in the last day should wake someone up. Breaking the silence is what exposes the rest, the stalled lock, the blocked snapshot, and the folded error. It requires no understanding of why, which is exactly why it works when your understanding is what broke.

Where this bites an AI workflow

Mine was a backup, which is easy to picture. The place this costs real money is an AI workflow, because those are built almost entirely out of signals like the one that failed me.

Take an agent that triages inbound requests overnight and reports what it handled. It runs on a schedule, it calls a model, it reads from a knowledge source, it writes results somewhere, and it tells you a number in the morning. Both failures are waiting in that sentence.

Silence first, because it hides better. Which of those steps, when it fails, produces an error you would see, and which produces an empty result that reads as a quiet night? A retrieval call that returns nothing because the index is stale looks exactly like a retrieval call that returns nothing because there was nothing to find. A model that declines to answer and a model that answers "no action needed" both arrive as no action. And an agent that never ran at all produces the calmest report of the week. That is the same fold, sitting inside the workflow instead of underneath it.

Saturation is the quieter waste. The morning number says forty-seven handled. Forty-seven does not separate the requests a human should have looked at from the ones that were always going to be routine, so nothing you do changes at forty-seven, or at sixty, or at twelve. You are paying to produce a number that no decision depends on, and reporting it as coverage.

The difference from my backup is timing. A backup fails visibly on the day you need it. An AI workflow can report success for a quarter, and the number it reports is one somebody is making staffing and spend decisions against.

The fix is the same three properties, applied one layer up. Every reported count separates the cases you would act on differently, or it is decoration. Every zero is distinguishable from could-not-check, which means each run leaves a receipt saying what it examined and not merely what it found. And something fires when the report simply stops arriving. None of that is model work. It is workflow ownership, and it is almost always the part nobody was assigned.

The operator test

Three questions for your next operations review.

Which decision does this number change? Take any figure on the standing report and ask which action would be different at a higher or lower value. If the honest answer is none, you are carrying it because you always have. That is the saturation test, and it costs nothing to run.

When a check reports clean, how do we know it observed anything? Not whether it ran, mine ran. Whether it came back with a reading rather than a shrug that got written down as good news. If the answer is that it would have alerted otherwise, ask what alerts when the alerter stops.

Who would notice if a report simply stopped arriving? Not arrived-and-looked-wrong. Stopped. In most organizations a report that vanishes is quieter than a report that says something alarming, which means the worst failure produces the least reaction.

The standard

Every number your operation acts on should carry an answer to one question: which of the states we act on does this tell apart?

For a number that fires on everything, the answer is none, and no amount of accuracy fixes that. For a number that reports clean, ask the second question: what would have made this come out differently? If nothing would, it was never watching.

A signal that cannot separate the states you act on is not oversight. It is decoration that costs money and buys confidence you have not earned.

If you want to run this against your own AI workflows, the exercise takes an hour and needs no tooling: list every automated workflow that reports a number to someone, and give each one two columns. What does a failed run look like on the report, and which decision does the number change? The ones with a blank in either column are where I would start, and the ones blank in both are where I would start today. That list is the beginning of a failure-mode heatmap, and you do not need me to make the first pass.

If the blanks expose a workflow nobody fully owns, that is the work my agency does: the Readiness Scan turns that list into a failure-mode heatmap, an ownership and handoff plan, and a 30/60/90 rollout roadmap. Work with OIA on one workflow.

No new backup was written for two nights and the check said fine, because fine was the only thing it could say.