When the Fix Never Reaches the Customer

A correction absent from a workflow's reuse decision leaves that workflow able to accept an old result as current.

A customer can receive the old answer after your team fixes it.

Consider a support workflow that drafts replies from an approved reference document. A customer finds a wrong instruction. The team corrects the document, checks the new wording, and closes the ticket. Next morning, an agent resumes a saved draft and sends the instruction everyone thought they had removed.

That is a hypothetical example, but the ownership problem is concrete. The head of support owns the promise that the correction took effect. The person editing the document owns a smaller task. Between those two sits the work already generated, saved, or waiting to resume. If the correction never reaches it, the customer spends time reporting the same error and the team spends time investigating a ticket marked resolved.

The question to ask when someone says "fixed" is which result they changed.

The run that did not need to run

I met the underlying mechanism in an internal video workflow. A caption extended beyond the end of the picture. The quality check refused it, and a subtitle sampler tried to read beyond the video's end. This was a local build; the execution record says the upload had not run.

The caption rule was corrected. The saved captions still survived a rerun.

The regression test records why: the stage resumed from its completion receipt instead of transcribing again. A completion receipt is a record of the earlier run and the inputs, settings, and output files associated with it. The reuse decision could see evidence that the old work was complete. It could not see the relevant change to the rule that produced it.

The correction sat behind a branch the program no longer entered.

This is an awkward failure to investigate because restarting the workflow looks like the obvious next step. You invoke it again, it finishes quickly, and the original problem remains. The new code has nothing to say about that result. It was skipped.

A true receipt for the wrong question

There is nothing inherently wrong with keeping work that has already succeeded. Transcription, rendering, retrieval, and model calls cost time. Resuming a long job without repeating every completed step is useful behavior.

The difficulty is deciding what "the same job" means.

If the source media, transcription model, and spelling corrections match, the saved captions look reusable. Change the rule for trimming captions to the video's duration, however, and those matches no longer answer the whole question. The output was produced under a rule that has changed.

A file hash can establish that the file has not changed. That is valuable evidence. When the file contains the old error, an unchanged hash also confirms that the old error is still there. Whether that file is suitable for use now depends on more than its integrity.

The same distinction appears in ordinary build engineering. Bazel's remote-caching documentation describes actions through their declared inputs, commands, environment, and outputs. Reuse is organized around an identified action and its result. That is a useful precedent for AI workflows: define the work precisely enough to know when a saved result belongs to it.

The word "declared" carries the burden. A dependency omitted from the comparison has no way to invalidate the result through that comparison.

Make the change visible to the reuse decision

The caption fix added the normalization rule's version to the configuration checked before resuming. The caption stage supplies its input files, model, spelling-correction configuration, and normalization version. The receipt comparison checks that configuration alongside the recorded files.

The regression test exercises both sides. Run the stage twice with the same rule and it calls the transcriber once. Change the rule's version and run again: the transcriber is called a second time, and the receipt's configuration hash changes.

Those assertions describe the regression test in the committed source. They are a narrow check of invalidation, not evidence that every dependency in the whole pipeline has been captured.

There is still a human obligation in this implementation: the normalization version is maintained by hand. A relevant rule change needs a version bump. Forget that bump and this particular signal cannot help. Versioning an identified dependency is progress; it does not make dependency discovery automatic.

For the hypothetical support workflow, the equivalent is deciding which version of the reference document a saved answer depends on. An answer generated against one approved revision cannot silently inherit the status of a later revision just because both live at the same document URL.

That dependency might be represented by a content digest, an immutable revision identifier, or another stable identifier supplied by the system that owns the reference. The requirement is observable: a relevant correction must reach the decision that would otherwise reuse the saved answer.

Rebuilding is only part of the correction

Invalidating an intermediate result is not the end of the path. A corrected set of captions still has to reach the video someone watches. A refreshed reference document still has to reach retrieval, generation, any saved draft, and the message selected for delivery.

Some of those stages need to run again. Others need to stop using the old result. The important question is which downstream outputs depend on the thing that changed, and what the workflow does with each of them.

This is where the workflow owner has work that no individual component owner can finish. The person responsible for the document can show the corrected revision. The person responsible for generation can show a new answer. The person responsible for support must establish which answer is about to leave the queue.

Do that check at the last point where a stale answer can still be prevented from reaching the customer. Compare the answer's recorded dependencies with the version approved for that delivery. If they disagree, hold that answer and regenerate or route it for review. If the required version cannot be established, the uncertainty belongs in the delivery decision.

The scope of that hold matters. A corrected instruction for one workflow does not, by itself, justify stopping unrelated work. A useful gate names the affected output, the mismatch, and the next action the operator can take.

New evidence can produce the same bytes

It is tempting to prove the fix by showing that the output changed. Sometimes it should. Sometimes the corrected rule reaches the same answer for the input in front of it.

The later caption revision illustrates this limit. It tightened the boundary to whichever ended first, the narration or the rendered picture. The recorded rerun produced identical output because those two durations coincided for that revision. The rule had been reevaluated; different bytes were not required.

That distinction changes the evidence you ask for. You need to see that the affected stage used the intended rule and that its result satisfies the requirement. A changed hash alone establishes neither. An unchanged hash alone does not disprove either.

For a support correction, choose a request that actually depends on the corrected instruction. An unrelated question returning the same answer tells you very little about whether the correction propagated.

Reopen one corrected ticket

There is a useful exercise for the next time your team closes an AI-related defect. Take the example that exposed it and repeat it through the normal reuse path in a test environment. Keep the saved draft or completed job that existed before the change. An empty environment can demonstrate the new logic while never exercising the old result that would have been reused.

Before making the change, identify the expected difference for this example. Then apply the correction through the normal process. Observe whether the old result is reused, invalidated, or explicitly retained under an approved older version. Follow the chosen result far enough to inspect what the intended user would receive.

Also repeat the unchanged case. If every request rebuilds everything, you have avoided stale reuse by removing reuse. That might be a reasonable temporary response, but it is a different operational tradeoff, with a cost the owner should understand.

This exercise investigates one dependency and one path. It cannot certify all the caches, saved jobs, references, and downstream systems a business uses. Its value is more specific: it shows whether a correction reaches the part of the workflow most likely to preserve the old result.

Decide what happens to work already in progress

The newest revision is not automatically the right revision for every job. Some work needs to finish under the instructions approved when it began. Reproducibility and rollback depend on retaining those versions.

That is a decision to make explicitly. Record which jobs keep the older version, who authorized that treatment, and which outputs must be withdrawn or rebuilt. An older result intentionally retained for a defined job is different from an older result accidentally presented as corrected.

For consequential actions, check that decision immediately before execution. Bind the decision to the exact output and its applicable approval; if either changes before the action occurs, require a new decision. A job started before a correction and delivered afterward crosses the very boundary the edit was meant to change. The starting version alone cannot tell you whether it is still approved to finish.

This does not require asking the model whether the correction is safe. The model can help produce a revised answer. The workflow still needs a deterministic way to identify the applicable version, control reuse, and hold an output when the required evidence is missing.

The owner of the correction owns its reach

Last week's edition asked whether a generated document's claims had actually been checked. A checked correction still has another journey to make: from the place it was edited to the place the business relies on it.

That journey deserves an owner. Someone has to know which saved results depend on the change, what invalidates them, which older versions remain approved, and where to observe the result. Closing the source ticket covers only part of that work.

If a correction in your operation keeps coming back, send me a LinkedIn message naming the workflow, what changed, and where the old result still appears. Those three details are a useful place to start investigating.

The next time a team says an AI workflow is fixed, ask to see the corrected result through the path that used to reuse the broken one.