Dan Mercede

The Recon Refuted the Task: Building Infrastructure With an Agent You Can Trust

Sat, 11 Jul 2026 07:00:00 +0000

I asked an agent to build a monitor: something to watch the background jobs across my servers and tell me when one broke. It came back and told me not to build it.

Not out of laziness. It had spent its first hour running probes against the live machines instead of writing code, and the probes said the thing I asked for already existed five times over. One platform watched the hosts. A container metrics collector watched the containers. The init system watched its own units. Two more scripts watched the edges. What I was actually missing was not another detector. It was the layer that reads five noisy signals and says "these three are the same incident, here is the likely cause, here is the runbook that fixes it."

That refusal was the single most valuable thing the agent did on the whole project. A detector I did not need would have shipped in an afternoon, run green forever, and watched nothing that mattered. The hour of recon that killed it is the reason I trust what got built instead.

This guide is the anatomy of that trust. It is worth nothing that an agent seems careful. What matters is the specific, checkable structure that let this one build a piece of production monitoring I now depend on: a recon step that argues with the task, a finish line a model can actually verify, a plan I approved before any code existed, autonomy bounded by construction rather than by a polite prompt, and acceptance by firing every control instead of watching it sit green.

Who this is for: anyone pointing a coding agent at real infrastructure, especially self-hosted systems where a quiet failure hides for weeks and "it ran without erroring" is not the same as "it works."

Why should an agent's first job be to refute your task?

In a 2026-07 build, the recon phase refuted the premise of the task before a line of code existed. I had asked for a holistic watcher because "nothing watches the jobs holistically," and a live sweep found five watchers already running. Four background units were failing at that exact moment, invisible because the watcher's coverage list was fixed at install time and never grew as new units shipped.

This is the crown rule of pointing an agent at infrastructure: recon is not information-gathering, it is error-correction. A first draft of any plan is full of confident, wrong premises, because the person writing the task does not have the live state in front of them. The job of the recon step is to take each premise and try to break it against the real system. "Nothing watches this" breaks the instant you list what is watching it. "Add a host-metrics exporter to one of the nodes" breaks when you find that node already runs a container metrics collector and ships its host metrics through a separate platform, so your exporter would double-count and page on itself.

The failure mode to design against is the opposite: an agent that takes the task at face value, builds exactly what you said, and hands you a clean diff for the wrong thing. A clean diff describes the code, not the world it runs in. Those four silently-failing units were the proof. Every one of them had been shipped after the monitor's coverage list was written, and the list was a static allowlist that rots the moment the estate grows. The highest-value item on the whole project turned out to be a watcher-of-the-wiring, an audit that asks whether newly-shipped units are actually covered. I never would have thought to ask for it. The recon found it by refusing to believe the task.

What makes a finish line a model can actually check?

The eleven files this run produced were never seen by the thing that graded it. A small, fast model scored the finish from the transcript alone, never running a command or opening a file. That one constraint rewrites how you write a done condition. "The eleven files exist and the tests pass" is unverifiable to that grader, so the run either loops forever or passes on the agent's word. "One final turn shows fresh command output and a self-audit table" is checkable in a single read.

An autonomous agent needs a terminal condition, the signal that tells the loop it is finished. The naive version describes the world: files present, tests green, service deployed. The problem is that whatever checks the condition is usually another model, and it has no eyes on your disk. It sees the transcript. A done condition phrased as a fact about the filesystem is therefore either uncheckable, so the run never converges, or taken on faith, so the agent can declare victory while nothing works.

The fix is to make the finish line transcript-demonstrable. Require the run to surface, in one final turn, the fresh output of real commands: the file listing, the diff, the passing test log, the grep that proves the wiring is live. Then require a self-audit table that maps each acceptance criterion to the evidence for it. Now the grader has something to grade. This is not ceremony. It is the difference between a run that ends because a model felt finished and a run that ends because it showed its work. Write the condition from the grader's real capabilities, not from a template you copied without asking what the grader can see.

Why gate the plan instead of the finished code?

Before any implementation, the recon collapsed nineteen candidate work-items into a design with exactly four genuinely-open decisions, and I resolved all four in a single structured prompt. Zero of the nineteen items collided with a project already in flight. The approval gate sat on the design, in the middle of the arc, not on the finished pull request at the end, because the middle is the only place an approval is cheap enough to be honest.

There is a strong pull toward letting an agent run start to finish and reviewing the pull request at the end. For infrastructure, that is the most expensive possible place to learn the plan was wrong. By the time a diff exists, the agent has committed to an architecture, and your choices are approve-with-misgivings or throw away real work. Reviewing a plan costs minutes. Reviewing and then unwinding a wrong build costs a day.

So the arc has one hard human gate, between research and implementation, and it is the only step that requires me. The agent produces a map: what it will build, what it deliberately will not, every prior decision it reconciled against, and a short list of the forks it genuinely cannot resolve alone. I answer the forks. Everything already settled is asserted plainly so it does not get re-litigated, and only the real decisions reach me. This is also where scope discipline lives. Nineteen items were dispositioned explicitly, each exactly once, so nothing was silently dropped and nothing raced work already running elsewhere. The plan is the artifact you can still change cheaply. Gate there, not at the end.

How do you make autonomy safe by construction rather than by prompt?

The remediation layer runs in three tiers, and none of them can take an action, because the capability to act was never written into any of them. The triage model sees only captured text and is never handed a tool it could call. The advice layer emits a proposal into a chat thread and holds no code path that executes. The incident-filing step can open a draft but has no path that closes. Each limit is structural, so a jailbroken or simply confused model hits a wall, not a moment of discretion.

"Propose-only" written in a system prompt is a request, and a request is not a safety property. The version that holds is the one where the capability is absent. The triage step is the clearest case. Instead of an agent with shell access reasoning about my servers, it is a deterministic collector that captures read-only output and a model that receives that text and returns structured JSON. The model cannot run anything, because nothing runnable was ever put in its hands. Its blast radius is the size of a JSON object.

The same shape repeats up the stack. The layer that suggests a fix writes the suggestion to a thread for a human to read; no branch anywhere in it executes the fix, so no prompt can talk it into one. The step that records an incident can draft a writeup, and the rule that an incident closes only after a human has confirmed the root cause and verified the fix is enforced by that step having no close path at all. This is graduated autonomy done honestly: each tier holds exactly the capability it has earned, and the tiers above it are not one clever sentence away. When you catch yourself trusting a model because you told it to behave, you are holding a prompt, not a gate.

Why isn't "armed and green" the same as working?

Every control that shipped was accepted by firing it, never by watching it sit healthy. We injected a synthetic fault and watched it travel the whole path to a chat thread. We ran one real triage cycle end to end and read the verdict it produced. We handed the refusal paths a case they should reject and confirmed they rejected it. A monitor that is armed and green has told you nothing at all until you have made it catch something.

The most seductive lie in operations is the green dashboard. An alerting rule with no alert, a remediation daemon with a healthy badge, a heartbeat that has never missed: all of them look identical whether they work or are quietly dead. "Armed and green" is a claim about a control's own status, not about whether it does its job. I have been burned by a reaper that reported healthy for weeks while reaping nothing, because nothing had ever forced it to act and reveal that it could not.

So the acceptance bar for the controls here was a fired proof. For the coverage audit, ship a unit it should flag and confirm the flag. For the triage path, run a real cycle and read the result. For the propose-only guarantee, feed it a case that must be refused and confirm the refusal returns an error instead of an action. The heartbeat is the honest exception worth naming: rather than trust a new bespoke alert, wire it onto the estate's already-proven dead-man monitor, the same one that pages when any other producer goes silent, so it inherits a firing that was tested long ago. A control you have watched catch a real event is a fact. A control you have only watched stay green is a hope wearing the costume of a fact. The extra hour spent writing synthetic faults is the cheapest insurance you will buy on infrastructure, because the alternative is discovering the control was decorative during the incident it existed to catch.

What do the five checkpoints add up to?

Across the project, five practices did the load-bearing work, and each one guards against a different way agent-built infrastructure fails silently. Not one of them depends on the model being smart. Every one of them is structure the human keeps: an argument before the build, a finish line a grader can read, a gate on the plan, capability bounded by construction, and acceptance by firing the control.

Five checkpoints between a task and production. Each catches a failure mode the others are blind to; remove one and that class of silent failure ships.

Checkpoint	The silent failure it prevents
Recon that refutes the task	building the wrong thing, cleanly
Transcript-demonstrable finish line	a run that ends on the model's own word
Approval gate on the plan, not the PR	learning the architecture was wrong after the work is done
Autonomy bounded by construction	a prompt talked into taking an action
Acceptance by firing the control	a decorative monitor that is green and dead

The through-line is that trust in agent-built infrastructure is not a feeling you develop about a capable model. It is a property of the harness around it. The model in this arc was the same model that wrote the confident, wrong first draft of the plan. What made its output trustworthy was that every step had a place where a wrong assumption got caught by something other than the model's own judgment. Recon caught the wrong premise. The grader caught the unfinished run. I caught the wrong plan. Construction caught the unsafe action. A fired test caught the decorative control.

Point a fast writer at your infrastructure with none of that, and you get exactly what the speed implies: the wrong thing, built well, running green, watching nothing. Keep the five checkpoints, and the agent becomes what it should be, a very fast pair of hands on a short and very well-lit leash. That is not a limit on what agents can do for operations. It is the thing that makes handing them the keys a sane decision instead of a hopeful one.

The Router Is a Trap: Running Other Models Under Claude Code, Off Their Own Subscriptions

Wed, 08 Jul 2026 07:00:00 +0000

I wanted a simple thing. Keep a Claude model orchestrating inside Claude Code, but when it fans out to subagents and background jobs, run that work on other models paid for by other subscriptions, so it stops drawing down my Anthropic allocation. The obvious build is a router: point Claude Code at a proxy, let the proxy translate each call to Gemini or GPT or a local model. That is the popular path, and it is a trap.

Two findings flipped my plan. The router path corrupts tool calls badly enough to be useless for agentic subagents. And the constraint that actually decides the architecture is legal, not technical: reusing a consumer subscription inside a third-party harness is prohibited and actively enforced at three of the four vendors I cared about. What survives both problems is the unglamorous option I already had running.

Who this is for: anyone trying to cut their Claude Code bill by offloading subagent or workflow fan-out onto other models, and anyone about to install a community "just point it at this proxy" bridge without reading what it does to your sandbox.

What are you actually trying to buy?

The goal is narrow and worth stating before the tooling: keep the frontier orchestrator (a Claude model) as Claude Code's primary driver, and route only the fan-out (subagents, workflow agent() calls, background tasks) to other models on their own billing. A Claude Code Max plan meters a weekly token allocation that resets on a fixed cadence, so every subagent you can move off that pool is allocation you keep for the orchestration that has to stay on Claude.

The key word is off-budget. This is the test every option below has to pass. Running a cheaper Claude model as a subagent does not pass it: Claude Code's native per-subagent selection is Anthropic-alias-only, and a Haiku or Sonnet subagent still bills the same Anthropic pool. Saving per-token cost is not the same as taking work off the allocation. Only routing to a genuinely external provider does that, which is why the interesting question is how to reach a non-Anthropic model at all.

Why does the router path corrupt your tool calls?

The most-starred tool in this space, claude-code-router, sits at roughly 35,600 GitHub stars as of mid-2026, and it works by exploiting one documented fact: Claude Code does not validate model IDs behind a custom ANTHROPIC_BASE_URL. Anthropic's own model-config docs say the ID check "runs only on the Anthropic API," so behind a gateway your provider defines the names and Claude Code passes any string through unchecked. That single gap is what every under-the-harness router is built on.

It works at the transport layer and fails at the semantic one. The dominant, recurring failure is tool-calling corruption. When a non-Claude model runs behind Claude Code's Anthropic-shaped API, the translation shim mistranslates tool calls: optional parameters get injected as empty strings (a Read arrives with pages: ""), which Claude Code rejects, "producing an infinite loop that burns reasoning tokens until the user manually interrupts." Server tools like web search get mistranslated into ordinary functions, so search comes back empty or misleading. Some local-model adapters emit plain text that merely mimics a tool call and then does nothing with it. On top of that, documented proxy layers drop MCP server tools, ignore prompt-cache control, and turn parallel tool use into sequential calls.

For a chat turn, that degradation is survivable. For an agentic subagent whose entire job is to call Read, Edit, Bash, and MCP tools reliably, it is disqualifying. The proxy gives you the model behind the endpoint and takes away the tool loop that made the endpoint worth reaching.

Why is the Terms of Service the real constraint, not the tooling?

Here is the finding that actually decided the architecture: as of mid-2026, reusing a consumer subscription inside a third-party harness is prohibited and enforced at three of the four vendors, so the tooling question is downstream of a legal one. The enforcement is not theoretical, and in one case it is not even loud.

Anthropic made the prohibition explicit in a February 2026 consumer-terms revision (tokens permitted only in Claude Code and Claude.ai), and on April 4, 2026 blocked third-party harnesses from using Claude Max subscription rate limits outright. Bring-your-own API keys still work; subscription-OAuth reuse does not.
Google is the cautionary tale. It deployed abuse detection and silently banned paying AI Ultra subscribers (a $249.99-per-month plan) who drove third-party CLIs on their consumer OAuth in mid-April 2026, with reports of the ban cascading to Gmail and Workspace. It then stopped serving consumer and Pro or Ultra tiers to its IDE code-assist extensions on June 18, 2026. If your estate leans on Google Workspace, an OAuth-reuse ban is not a coding-tool outage, it is a Gmail outage.
xAI's terms bar automated, non-human (bot or script) access to Grok and reserve suspension at sole discretion; the only sanctioned programmatic path is the separate pay-per-token API.
OpenAI is the outlier. It publicly sanctions reusing a ChatGPT subscription in third-party tools with "the same usage as Codex," and Codex-into-third-party has an official adapter. Subscription Codex is governed by the consumer ChatGPT terms and metered as plan usage, not raw API billing.

The net for the original dream of "route all my subscriptions through one proxy": it is durable for exactly one of them, GPT via Codex. Every other subscription seam rides reverse-engineered OAuth that the vendor is actively closing. One corroborating datum: a Claude-Max-as-API seam that made the rounds earlier in 2026 survived about two months before the vendor shut it. Treat any subscription-brokering feature as a liability with a short shelf life, not a capability. For Gemini or Grok specifically, use a paid API key, never the consumer login.

What actually works: shell out to headless CLIs

The mechanism that clears both the tool-fidelity trap and the Terms-of-Service wall is the one I already ran in production for months: keep Claude orchestrating and shell out to each other vendor's headless CLI as a subagent. Instead of translating Claude Code's API to another provider, you invoke codex exec, gemini, or grok -p as a subprocess, each authenticated to its own login, each doing its own native tool calling.

This inverts every problem the proxy created. There is no Anthropic-to-OpenAI translation shim, so there is no empty-string-parameter loop and no mistranslated web search; each CLI speaks its own protocol natively. The work runs on that CLI's own credentials, so a Codex call bills the ChatGPT plan and a local-model call bills nothing, both off the Anthropic allocation. And it matches a doctrine most mature harnesses already follow, which is to prefer a working CLI over a bespoke integration: a post-push review pipeline that drives Codex headless is the same pattern, just pointed at review instead of fan-out.

The tradeoff is honest. You give up the illusion of one unified /v1 endpoint and instead orchestrate N processes, joined by your own wait-and-collect logic. In exchange you get native tool loops, per-vendor billing, and no dependency on a subscription seam that gets patched out from under you.

How do you keep the delegation layer from bypassing your sandbox?

The security catch is real and specific: nearly every community CLI-bridge I looked at launches its child CLI with a permission-or-sandbox bypass flag baked in (--dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox, --yolo, -y). That is convenient and it is also how you hand an untrusted subagent's output the keys to your workstation. When I shipped my own delegation layer in July 2026, the rails were the point, not an afterthought, and they are enforced in code with tests, not just documented.

Five rails carry the safety, and each is one small function:

Never emit a bypass flag. A denylist is asserted in a unit test so a bypass flag can never regress into the spawned argv. The read-only sandbox is the default; write access is an explicit, separate opt-in.
Scrub the child environment by whitelist, not denylist. The child gets PATH, HOME, and its own provider key, and nothing else. A future secret-shaped variable you have not thought of yet does not survive by default, because the default is to drop, not to keep.
Resolve the CLI by absolute path. Never exec a bare command name; resolve it to an absolute path first so a planted binary earlier on PATH cannot shadow the real one.
Close stdin. Recent Codex versions deadlock on a non-TTY pipe, so the child gets stdin pointed at /dev/null.

# read-only by default; workspace-write is an explicit, separate opt-in
sandbox = "workspace-write" if write else "read-only"
argv = [resolve_cli("codex"), "exec", "-s", sandbox, task]
assert not any(flag in argv for flag in BYPASS_FLAGS)  # tested, not just asserted here

The fifth rail I only found by running the thing, which is the reminder that live verification beats reasoning. Grok's CLI needs an explicit headless flag: grok -p "<prompt>" prints the answer and exits, but a positional prompt with no flag opens the interactive TUI and hangs a non-interactive caller. A positional prompt cost me a two-minute timeout before I read the help text. The regression is now pinned by a test that asserts the -p flag is present in the built argv.

# a positional prompt opens grok's interactive TUI and hangs a non-TTY caller;
# -p / --single prints and exits.
argv = [resolve_cli("grok"), "-p", task, "--output-format", "json"]

What should you actually build?

If you take one architecture from this, take this shape, which I settled on after two research runs and a working implementation in July 2026: Claude orchestrates, and it delegates to a small, gated set of headless CLIs chosen by Terms-of-Service durability first and cost second.

Orchestrator: a Claude model in Claude Code, unchanged.
Primary off-budget subagent: GPT via codex exec on the ChatGPT plan. This is the one durable subscription-reuse path, so it earns the default slot.
Cheap bulk fan-out: a metered API key (DeepSeek and Kimi run roughly an order of magnitude cheaper than frontier Claude on input tokens) or a free local model on your own GPU box via a local inference endpoint. No subscription to reuse means no ban risk.
Gemini or Grok, if at all: only on a paid API key, isolated from the primary Google account. Never the consumer login.
Mechanism: headless-CLI subprocess, not a proxy. If you ever want a proxy for API-key models, the routers are fine; just treat their subscription-brokering as the liability it is.

The three paths compared, so the choice is legible:

Feature	Router proxy	Headless CLI (recommended)	Native per-subagent
Keeps Claude orchestrating	Only if you patch aliases	Yes, cleanly	Yes
Routes subagents to other providers	Yes, but Claude-unaware	Yes, per call, model named	No, Anthropic aliases only
Off the Anthropic budget	Yes	Yes	No, still bills the same pool
Tool-calling reliability	Poor and brittle	Native per CLI	Native
Subscription legality	Rides the prohibited seam	Depends on vendor (GPT sanctioned)	Not applicable

The plan I walked in with, a single elegant router, was the wrong instinct dressed up as the clean one. The winning design is a handful of subprocesses behind a security gate, choosing each model by whether its vendor will still let you in next month. Boring, durable, and off-budget. That is the trade, and it is a good one.

The vendor terms and enforcement dates in this guide are time-sensitive and were accurate to mid-2026; enforcement is active and moving, so re-verify a vendor's current terms before you build on any subscription seam.

Why Most AI Governance Starts Too Late

Tue, 07 Jul 2026 07:00:00 +0000

Policies, dashboards, and audit logs make AI systems observable. They do not make them governable. Once an agent can change state, governance has to stop the action before the mutation - not explain it afterward.

Most AI governance starts after the action.

A model selects a tool. An agent executes a workflow. A database is updated. A message is sent. A file crosses a permission boundary. A business process advances.

Then the system logs it. Then a monitor evaluates it. Then a dashboard reports it. Then a human reviewer decides whether the action should have happened.

This is not governance. It is after-action accounting.

The distinction matters because the systems now being deployed into enterprises are no longer passive prediction services. They are execution systems. They call tools, move data, update records, trigger workflows, produce decisions, and operate across persistent context.

When an AI system can mutate state, the decisive question is not whether the organization can explain what happened afterward. The decisive question is whether the system could have been stopped before the state changed.

Most AI governance architectures answer that question too late. This article argues for moving the control point - from after-action review to pre-execution authority - and lays out what that shift requires in practice: where enforcement has to sit, what artifact it has to produce, and how to test whether a given architecture actually governs or merely observes.

The category error

Many organizations still treat AI governance as an organizational layer. They create policies, define responsible-AI principles, assign risk owners, build dashboards, add logging, monitor outputs, run evaluations, and create escalation paths.

These controls are useful. They are not sufficient.

The category error is treating governance as a visibility problem when the actual failure mode is an execution-control problem. Visibility answers a retrospective question - what happened? Governance has to answer a different one - what is allowed to proceed? Those are not the same system.

A log records an event. A dashboard visualizes an event. An alert reacts to an event. A policy document describes intended behavior. An evaluation estimates behavior under test conditions. None of those mechanisms, by themselves, prevents a disallowed action from completing. They observe, classify, and inform. They do not govern.

Governance begins when execution depends on authority.

State mutation changes the problem

The traditional model-risk frame assumed a relatively contained system. A model generated a score. A human interpreted the score. A downstream system acted separately. Validation could occur periodically, and governance could sit around the model.

That assumption breaks when AI systems become agents. An agent does not merely produce an output. It forms an intent, chooses a tool, passes parameters, invokes an API, retrieves context, writes data, and continues operating. The risk is no longer whether the model's answer was accurate. The risk is whether the system can perform an unauthorized action.

A state mutation is any durable or externally consequential change made by the system - a database update, a payment instruction, a CRM change, a permission change, a document transfer, a message sent to a customer, a workflow approval. Once the mutation completes, governance is downstream. At that point the system is no longer deciding whether the action should occur; it is deciding how to explain, remediate, or document the action that already occurred. That is a fundamentally different control problem.

The problem with post-hoc governance

Post-hoc governance accepts a dangerous premise: that unauthorized action may occur before intervention. This premise is often hidden inside otherwise reasonable architecture - the system executes, the log records, the monitor detects, the alert fires, the reviewer investigates, the organization remediates.

That sequence can support incident response. It cannot serve as the primary enforcement model for high-risk AI execution, because the latency is the failure:

If an AI workflow sends regulated data to the wrong endpoint, the alert does not undo disclosure.
If an agent modifies a production record, the log does not prevent the mutation.
If a system approves an action outside policy, the dashboard does not restore authority.
If a tool invocation uses authorized credentials with unauthorized parameters, post-hoc review does not change the fact that the system was allowed to act.

Governance evaluated after execution is telemetry. Telemetry is not enforcement.

The missing unit is the execution boundary

The central unit of AI governance is not the model, the prompt, the policy document, or the dashboard. It is the execution boundary - the point where intent becomes action.

Before that boundary, the system can reason, propose, retrieve, rank, summarize, or plan. After that boundary, the system changes something. Governance has to sit at that boundary - not near it, not parallel to it, not downstream from it.

At the execution boundary, the system must answer a small set of deterministic questions before the action proceeds:

What is the system trying to do?
Who or what authorized it?
Which policy applies?
Is the requested action inside scope?
Are the parameters authorized?
Is the current context still valid?
What evidence proves the decision?
What happens if the answer is ambiguous?

If the system cannot answer those questions before mutation, it is not governing execution - it is letting execution run ahead of governance. That is the structural failure.

Authority must be evaluated before mutation

Authority is not the same thing as capability. A system may be capable of calling an API; that does not mean it is authorized to call it. A system may be capable of sending an email; that does not mean it is authorized to send this email, to this recipient, with this content. A system may be capable of updating a record; that does not mean it is authorized to update this field, in this account, under this workflow, at this time.

Most AI systems collapse capability and authority. They grant tools to the agent and rely on prompts, policies, or application logic to keep behavior inside bounds. That is weak architecture. The model should not be the enforcement layer for its own behavior.

A governed system separates intent generation from execution authority. The AI system can propose an action. A control plane decides whether the action can proceed:

intent = agent.propose_action(context)

decision = authority_gate.evaluate(
    intent=intent,
    policy=active_policy,
    actor=session_identity,
    context=current_context,
)

receipt = receipt_ledger.write(
    intent=intent,
    decision=decision,
    policy_version=active_policy.version,
    actor=session_identity,
)

if decision != "permit":
    halt_execution(receipt)
else:
    execute_action(intent, receipt)

The important detail is not the syntax. It is the sequence. The action does not execute first. The log does not arrive later. The reviewer does not become the first real control point. Authority is evaluated before mutation, a receipt is created as part of the execution path, and ambiguity halts the transaction. Execution depends on governance.

Fail-open is the default failure mode

Many systems fail open without saying so. A policy service times out, so the workflow proceeds. A classifier returns an uncertain score, so the action routes anyway. A guardrail fails to parse the response, so the application retries. A logging service is unavailable, so the transaction completes without evidence. A human approval step is skipped because the system treats it as asynchronous.

These are not operational edge cases. They are governance failures.

A governed system must fail closed. Fail-closed means execution halts when authority cannot be verified, when policy cannot be resolved, when context is incomplete, when the requested parameters exceed scope, or when the evidence artifact cannot be written.

This is uncomfortable for product teams, because fail-closed systems interrupt workflows. That interruption is the point. Governance that cannot stop execution is advisory. It may be useful, informative, even necessary for audit review - but it is not enforcement.

Logs are not receipts

Audit logs are often treated as proof of governance. They are not. Logs are observational artifacts: they record that something happened. Receipts are enforcement artifacts: they prove that a specific action was evaluated under a specific policy, by a specific authority, at a specific time, before execution proceeded.

The distinction is mechanical. A log can be written after the fact; a receipt must be generated in the transaction path. A log can describe an event; a receipt binds the authority decision to the mutation. A log supports investigation; a receipt supports non-repudiation.

A log says: this happened. A receipt says: this was authorized, under this policy, by this authority, before it happened.

For AI systems that mutate state, this distinction determines whether the organization holds audit-grade evidence or merely operational telemetry. If a mutation can complete without a receipt, it can complete without governance.

Evals are not control planes

Evaluations are useful. They measure model behavior under defined conditions, and they help teams compare systems, detect regressions, test prompts, and assess risk before deployment. Frameworks like NIST's AI Risk Management Framework formalize this organizational discipline, and reporting conventions such as model cards make a system's intended use and limitations legible [1][2].

But an eval is not a runtime control plane. An eval does not authorize a specific action, inspect every tool invocation, bind policy to a transaction, halt execution when authority is missing, or prove that a state mutation was permitted before it occurred. The same is true for red-team reports, policy reviews, benchmark scores, and model cards. They contribute to governance; they do not replace enforcement.

A system can pass evaluations and still execute an unauthorized action in production. This happens because production execution contains variables that test environments cannot fully capture: live context, changing permissions, tool parameters, user-specific state, workflow timing, data dependencies, and long-running agent behavior. The governance layer must operate where those variables exist. That means runtime.

Guardrails are not enough when capability remains

Guardrails often sit at the model or application layer. They filter prompts, classify outputs, block certain content, detect policy violations, and shape behavior. These mechanisms are useful for many classes of risk. They are not sufficient for state-mutating workflows when the underlying capability remains available.

If an agent has network access, credentials, tool access, and a route to production systems, the organization is relying on the agent not to misuse capabilities it already possesses. That is restriction - the capability exists but is blocked by a rule.

Capability removal is different. It means the execution environment does not possess the primitive required to perform the unauthorized action: no direct network route, no persistent credential, no raw socket access, no production database path, no tool invocation outside a mediated boundary. A system cannot misuse a capability it does not have.

This is why substrate design matters. The execution environment should be treated as untrusted compute. It should not be able to route itself around governance. The agent can produce intent; it should not independently possess the authority, credentials, and network pathways required to turn that intent into production state.

Drift is a governance problem across time

AI systems do not only fail at a single transaction. They fail over time. A workflow that behaved correctly on Monday can become unsafe by Friday because context changed, permissions changed, memory accumulated, prompts shifted, tools changed, or the model began operating near guardrail boundaries.

Per-action authority checks are necessary, but not sufficient alone. Governance must also constrain behavior across time - not as passive monitoring, but as enforceable containment. Increasing guardrail triggers, repeated near-denials, escalation loops, changes in tool-use frequency, semantic drift in task execution, and the width of authority windows all matter.

A system that had authority yesterday does not automatically have authority tomorrow. Authority should decay. Execution paths should not remain open indefinitely. Long-running AI systems require temporal constraints because risk compounds across sequences, not only individual actions. Monitoring watches that sequence; governance intervenes in it.

The three-question audit

Any AI governance architecture can be tested with three questions.

1. Where does enforcement occur? If enforcement occurs after execution, it is telemetry. If it occurs outside the execution path, it is advisory. If it occurs inside the transaction path before mutation, it can govern. The location of enforcement determines whether the system can stop the action.

2. What happens on failure? If policy cannot be resolved, does execution proceed or halt? If the authority service is unavailable, does the workflow continue? If the parameters are ambiguous, does the system route anyway? If the receipt ledger cannot commit, does the mutation still complete? The failure path reveals the real governance model: a system that proceeds under ambiguity is fail-open; a system that halts under ambiguity is fail-closed.

3. What artifact proves enforcement? A dashboard is not proof. A policy document is not proof. A model evaluation is not proof. An application log is not enough. The artifact must show what action was requested, what policy was evaluated, who or what authorized it, what decision was made, when it occurred, and whether execution was permitted or denied. Without that artifact, enforcement is inferred - and inferred enforcement is not audit defensibility.

What better AI governance looks like

A governed AI execution system has a different shape. The model does not directly mutate production state. The agent does not hold standing authority. The execution environment does not contain unnecessary capability. Tool calls are mediated through an enforcement boundary. Every state-mutating action is represented as a structured intent. Policy is evaluated deterministically before execution. Ambiguity defaults to denial. Material actions generate receipts. Behavior is constrained across time. Substrate access is limited by physical and infrastructural boundaries, not only model instructions.

This architecture does not eliminate AI risk. It changes where risk is handled. Instead of discovering unauthorized behavior after mutation, the system constrains execution before mutation. Instead of relying on logs to reconstruct events, it produces receipts as part of the execution path. Instead of trusting the model to obey policy, it places policy outside the model and inside the control plane. Instead of allowing capability and hoping it is not misused, it removes unnecessary capability from the execution environment.

This is the shift from advisory governance to runtime governance.

The practical boundary

Not every AI interaction requires the same level of enforcement. A summarization assistant reading public documentation does not need the same controls as an agent that can approve refunds, modify customer records, change permissions, execute trades, or move regulated data.

The boundary is state mutation. When AI produces text that a human can ignore, governance can be lighter. When AI changes external state, governance must be structural. The higher the blast radius, the less acceptable post-hoc governance becomes. For low-risk workflows, observation may be enough. For consequential workflows, execution must depend on authority. That is the practical line.

Governance has to move earlier

Most AI governance starts too late because it was built around the wrong operating model - one that assumes AI systems are things to be evaluated, monitored, and reviewed. That was adequate when AI systems produced recommendations. It is inadequate when AI systems execute.

Execution changes the control surface. Once an AI system can mutate state, governance has to move from policy documentation to execution architecture, from dashboards to transaction boundaries, from logs to receipts, from after-action review to pre-execution authority.

The question is not whether the organization has AI governance. The question is whether governance can stop the system before the state changes. If the answer is no, governance starts too late. And if governance starts after mutation, it is not governance. It is hope.

Dan Mercede is a systems architect focused on governed automation and runtime enforcement for enterprise AI. He writes about fail-closed system design, control planes, and the architecture of human-owned intelligence.

References

[1] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji and T. Gebru, Model Cards for Model Reporting (2019), Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19)

[2] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023), NIST AI 100-1

Giving Your Coding Agent Web Access Without Wrecking Its Context Window

Sun, 05 Jul 2026 07:00:00 +0000

Most coding agents get web access the lazy way: a fetch tool that returns raw HTML, a search tool wired straight into the main context, and a metered API behind both. It works in a demo. Then the agent reads three documentation pages, and half its context window is <div class="nav"> and inline analytics.

I spent a day hardening the web-access layer for my own agents and measured every option on two axes almost nobody ranks first: cost and rate-limit resilience. The result is a stack that is free or self-hosted for the load-bearing paths, plus one structural pattern that matters more than any single tool: keep raw pages out of the main context by design.

This guide is the version I wish I had at the start. Every number is from a public source or a run you can repeat.

Who this is for: anyone giving an autonomous agent (Claude Code, a -p headless pipeline, Codex, an in-house harness) the ability to search, read, and drive the web, and paying for the tokens.

Why does giving an agent web access wreck its context window?

Because raw HTML is mostly not content. In one measured fetch, the same page was 9,541 tokens as raw HTML and 1,678 tokens as clean markdown, an 82% reduction. Typical HTML-to-markdown conversion lands nearer 20 to 30%, but the direction is constant: you are paying to put boilerplate into the model's attention, then paying again on every following turn.

The waste compounds three ways. Context flood: a large docs page is roughly 25,000 tokens of input, and it stays resident for the rest of the conversation. Cost: search results and fetched content are billed as input tokens on the turn they arrive and every turn after, so a page you read once is re-billed until it ages out. Rate limits: point a fan-out of parallel agents at a metered search API and you hit 429 almost immediately. The lazy path fails on all three at once.

Context rot is the quiet one. Even on a large context window, recall degrades as the window fills. A page of HTML you never needed is not free just because it fit.

Raw HTML can burn 80 percent of an agent's context. Self-hosted search, extract-first fetching, and a summarizing subagent keep only the signal.

What are the three axes of agent web access?

Web access is not one capability, it is three, and they want different tools. Search (find the URLs), fetch (read a page without drowning in it), and browser automation (click, fill, scroll a live app). In a 2026-07 review of the field, the cheapest resilient answer differed per axis: self-hosted for search and fetch, local-and-free for the browser.

Conflating them is how people end up overpaying. You do not need a headless Chromium to read a blog post; an extract-first markdown fetch handles the large majority of retrieval. You do not need a metered search API to read a page whose URL you already have. Name the axis, then pick the cheapest tool that covers it. Reserve the expensive, stateful browser for genuine interaction: clicks, forms, infinite scroll, auth walls.

The rest of this guide takes the axes in order, cheapest and most rate-limit-resilient first.

What is the cheapest, most rate-limit-resilient search stack?

Self-hosting wins outright. A self-hosted SearXNG metasearch instance runs on a 5 to 10 dollar per month VPS, aggregates roughly 95 to 200 engines (Google, Bing, Brave, DuckDuckGo and more) behind a JSON API, and charges nothing per query. Put a Valkey or Redis cache in front of it, because individual upstream engines rate-limit a single client IP hard, and the cache is what absorbs that.

If you want a hosted API instead, rank by the free tier and the rate cap, not the marketing. Exa gives 20,000 free requests a month. Brave gives about 1,000 free queries a month but caps the free tier at 1 query per second, which breaks the instant you do parallel fan-out search, a common agent pattern. Perplexity Sonar and Kagi are metered with no meaningful free tier. Anthropic's native web_search is the priciest per call at 10 dollars per 1,000 searches plus token costs, though its companion web_fetch adds no per-fetch fee and ships a max_content_tokens cap worth using.

One honest caveat carried from the research: a "Serper plus a reader" stack was reported at about 11 dollars a month for 10,000 searches and 10,000 extractions, against roughly 96 dollars for a single-vendor API on the same workload. That specific figure rests on a single blog and should be read as directional. The ranking (self-hosted free, then Exa/Brave free tiers, then metered) is the corroborated part.

What is the cheapest way to read a page without flooding context?

Extract-first, markdown, in that order. The single biggest lever is converting a page to clean markdown before it reaches the model, which cut tokens 82% in the measured case above. Jina Reader is the cheap default: prefix any URL with the reader endpoint and you get markdown back, free at 20 requests per minute with no key, or 10 million tokens on a free key, and it is open-source so you can self-host it. Trafilatura is a self-hosted Python extractor with no API cost and strong extraction quality, though it will not render JavaScript.

For scraping proper, the same free-first ranking holds. Crawl4AI is Apache-2.0 with no per-page fees; your only cost is the box it runs on. Managed Firecrawl trades that for convenience at roughly 83 dollars a month on its standard tier. The crossover is volume: at low and mid volume the managed credits are cheaper than running infrastructure; past high volume, the self-hosted crawler wins. Pick the tool, but do the markdown conversion no matter which you pick. Handing a model raw HTML is the mistake; the extractor is incidental.

Why does a thin CLI beat an MCP server for agent web tools?

Because an MCP server costs context on every host, and a CLI costs almost none. As of 2026-07, Claude Code defers MCP tool schemas until a tool is invoked, so an MCP is cheapish there. But Codex has no such deferral: every MCP schema loads in full, every session, and a harness that already carries a dozen servers pays that tax constantly. A thin CLI you shell out to via Bash costs zero resident context until the moment you call it.

So for anything I want available on more than one agent host, I ship a CLI, not an MCP. I built three this way: a search CLI over a grounded-answer API, a live-social search CLI, and a page-reader over Jina. Each self-wraps its own secret retrieval (read the key from the environment, or transparently re-run under a secrets manager if it is not set) so the caller passes no credentials, and each takes a --json flag so an agent can parse it. One implementation, every host, near-zero context. The rule generalizes: prefer a working CLI over its MCP unless the tool genuinely needs a stateful session.

Browser automation is the one real exception. Playwright, driving a real browser, is stateful and does not reduce cleanly to a stateless CLI, so it stays an MCP or an in-process library. There, the free-and-local option still wins: Playwright is Apache-2.0, runs on your own machine at zero LLM cost, and hits about 98% reliability on deterministic scripts. Running Playwright's MCP in its CLI mode reportedly cuts token use about 4x versus the protocol mode. Metered cloud browsers exist (Browserbase, Stagehand's hosted tier) but the free tiers are thin (Browserbase caps the free tier at 1 browser-hour, 3 concurrent sessions, 15-minute sessions), so reserve them for scale you have actually hit.

How do you build the fetch tool, and why run it in a subagent?

Because the tool only wins if the raw page never reaches the main agent at all. The pattern is small: a fetch CLI that returns bounded markdown, invoked inside a subagent whose job is to read the page and return only the distilled slice the main agent asked for. The main context sees a paragraph, not 25,000 tokens. Even the markdown never lands in the parent conversation.

The fetch CLI itself is about 120 lines. The load-bearing parts:

import math, os, urllib.request  # the full CLI also imports json (for --json) and sys

READER = "https://r.jina.ai/"
# The default python-urllib User-Agent gets a 403 from the reader's bot filter.
# Send a real browser UA. This one line is the difference between 200 and 403.
UA = os.environ.get("WEB_READ_UA",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/126 Safari/537.36")

def fetch(url, key=None):
    req = urllib.request.Request(READER + url, method="GET")
    req.add_header("User-Agent", UA)
    req.add_header("X-Return-Format", "markdown")
    if key:                                  # optional, raises free-tier limits
        req.add_header("Authorization", f"Bearer {key}")
    with urllib.request.urlopen(req, timeout=60) as r:
        return r.read().decode("utf-8", "replace")

def budget(md, max_tokens):
    # A stdlib token estimate: ~4 chars per token. Label it estimated.
    est = math.ceil(len(md) / 4)
    if not max_tokens or est <= max_tokens:
        return md, est, False
    return md[: max_tokens * 4].rstrip() + "\n\n... [truncated]", max_tokens, True

Three details earn their place. It is keyless by default (the reader's free tier needs no key), so the tool works with zero setup and only reaches for a key to raise limits. It carries a token budget (--max-tokens), so a runaway page cannot blow the context even inside the subagent. And it emits --json so the calling agent gets structured output, not a wall of text. The browser User-Agent is the one non-obvious line: a live smoke test, not the unit tests, is what caught the reader returning 403 to the default python-urllib agent. Unit tests pass with a mocked network; only exercising the real endpoint surfaces that class of bug.

What should you actually adopt?

Start free and self-hosted, add metered tools only where you have measured a need. As of 2026-07 the stack I run is: self-hosted SearXNG behind a cache for search, a keyless Jina-backed reader CLI for fetch (run in a subagent so raw pages never hit the main context), a self-hosted crawler for bulk scraping, and local Playwright for the genuine browser work. Native provider search sits behind all of it as a metered last resort.

The through-line is not a specific tool, it is two rules. Convert to markdown before the model sees a page, which reclaims most of the token bill on day one. And offload the fetch to a subagent, so the raw content, even cleaned, is summarized down before it touches the conversation the main agent has to keep coherent. Everything else is a cost-and-rate-limit optimization on top of those two moves.

The takeaway

Giving an agent web access is easy; giving it web access that does not quietly bankrupt its context window takes two deliberate choices. Extract to markdown, and read in a subagent. Do those, rank every tool by free-tier and rate cap before you reach for a paid API, and prefer a thin CLI over an MCP for anything that is not a stateful browser. The lazy path costs you a large slice of your web-read tokens, up to 80% on a boilerplate-heavy page, and a 429 under load. The fix is a day of plumbing and a 120-line script, and it pays for itself the first time your agent reads the docs.

The Reviewer Blessed the Bug: Why Agent-Written Code Needs Layered Review

Sat, 04 Jul 2026 07:00:00 +0000

A security reviewer read my diff and signed off on a specific claim: the credential-scrubbing was correct, "encoded access/secret forms are distinct, no overlap mis-redaction." It was thorough. It traced every log path. It was also wrong.

One review layer later, a different reviewer took the same function, fed it two overlapping credentials, and printed the result: a partial secret sitting in the log line. The redaction had eaten part of one secret and left the tail of another exposed. Same code, same afternoon, opposite verdict.

That gap between "a careful reviewer certified this" and "an independent adversary broke it in one run" is the entire case for layered review. When an agent writes your code, and increasingly it does, no single review pass is a safety net. The passes have to be independent, because independence is the only thing that makes their blind spots not line up.

This guide walks the real arc: one small log-scrubber, four review layers, five defects, and the exact modality that let each layer catch what the others could not see. Every bug here is a shape you can copy into your own checks.

Who this is for: anyone shipping code an AI agent wrote, or building the review harness that gates it, especially around self-hosted infrastructure where a quiet failure hides for weeks.

What was actually being hardened?

In a 2026-07 pass on a self-hosted LLM-observability stack (Langfuse on top of ClickHouse, with backups pushed to a MinIO object store and an encrypted off-box replica via restic), I hardened one function: a log scrubber that strips credentials out of backup-failure messages before they reach a cron log. Roughly 20 lines. It is exactly the kind of small, security-adjacent code an agent produces in seconds and a human skims in seconds.

The scrubber matters because backup scripts fail loudly, and a failure message often contains the command that failed, which contains the credentials. If a nightly backup dies and dumps an S3 secret key into a log that ships to your aggregator, you have turned a reliability event into a credential-exposure event. The scrubber's whole job is to make that impossible.

So it is a good test case: tiny, high-stakes, and the sort of thing everyone assumes is too small to need real review. It went through four layers anyway. Here is what each one saw.

Why does live-environment recon catch what no diff can?

Before any diff review, I ran the backup once against the real host, and that single run exposed a defect no code reader could have found: the script selected its MinIO container with a fuzzy name match, and the host happened to run two MinIO containers. The match grabbed the wrong one, hit Access Denied on a bucket it could not see, and reported a false CRITICAL every night while the actual backup was fine.

No diff review catches this. The code was locally correct. docker ps filtered by a name substring returns the first match, and on a one-MinIO dev box that first match is always the intended one. The bug only exists in the shape of the live environment, where a second, unrelated MinIO happened to sort ahead of the one the backup wanted. The lesson is old and keeps being true: a passing local test and a clean diff describe the code, not the world it runs in. The fix was to select the container by exact name, defaulted in-script, so a fuzzy match can never wander onto the wrong instance.

Recon is a review layer in its own right. It senses runtime state, a modality a static reviewer cannot reach. If your agent-review harness is all diff-readers, it stays blind to an entire class of defect that only appears on contact with production.

What does a pre-PR reviewer fleet catch that a single reviewer misses?

Before the pull request opened, I ran three independent reviewers over the diff (adversarial, correctness, and security), and the security pass caught a redaction bug hiding in plain sight: the scrubber replaced each secret with a shell pattern substitution, s="${s//${SECRET}/<redacted>}", and the unquoted needle is treated as a glob. A secret containing a bracket expression like [ab] is read as a character class, fails to match itself literally, and passes through the log unredacted. A * or ? in the secret fails the other way, over-matching and redacting adjacent log text. Both are wrong; only one leaks, and you cannot predict which without knowing the secret.

The fix is one edit, quoting the needle so the match is always literal:

# unquoted: a bracket-expression secret leaks, and * or ? over-redacts
s="${s//${SECRET}/<redacted>}"
# quoted: the secret is matched literally, both failure modes closed
s="${s//"${SECRET}"/<redacted>}"

The value of the fleet is not three times the reading. It is three different readings. Point one reviewer at a diff and you get one model's priors about what is worth checking. Point three with distinct adversarial framings and their blind spots do not fully overlap, so the union of what they notice is strictly larger. This is the cheapest layer to add and the one people skip most, because a single "looks good" from a capable model feels like enough. It is not enough. It is one sample.

Why did the post-push layer catch what the security reviewer had certified?

With an access key of ABC and a password of ABCDEF, a post-push reviewer ran the already-certified scrubber on the line password=ABCDEF and got back password=<redacted>DEF: a partial secret in the log. The pre-PR security pass had checked this exact function for overlapping-secret bugs and signed off, "encoded access/secret forms are distinct, no overlap mis-redaction." A second model, running the code instead of reasoning about it, broke that certification in one run.

Here is the whole bug. The scrubber redacted each secret in sequence. If a shorter secret is a substring of a longer one, the shorter replacement fires first, mutates the line, and the longer secret no longer matches itself, so its tail survives:

# ACCESS='ABC'  RESTIC='ABCDEF'
s="${s//"${ACCESS}"/<redacted>}"   # 'password=ABCDEF' -> 'password=<redacted>DEF'
s="${s//"${RESTIC}"/<redacted>}"   # no longer matches; 'DEF' leaks

The fix is to stop treating the needles as an ordered list and redact them longest-first, so no shorter needle can fragment a longer secret before that secret is matched whole:

# sort the needles by length, descending, then redact each literally
if [ ${#needles[@]} -gt 0 ]; then
  readarray -t needles < <(printf '%s\n' "${needles[@]}" | awk '{print length"\t"$0}' | sort -rn | cut -f2-)
  for n in "${needles[@]}"; do s="${s//"${n}"/<redacted>}"; done
fi

Longest-first closes the substring case, where one secret sits wholly inside another. It does not close the rarer case of two distinct secrets that overlap in the log line, a suffix of one equal to a prefix of another; random credentials almost never do this, but if yours can, redact in a single combined pass instead of a loop. State the bound; do not sell the fix as total.

This is the load-bearing section, so sit with what happened. A competent reviewer did not skip the overlap question. It considered the overlap question and answered it wrong, because it reasoned about the encoded forms being distinct and missed that the raw forms could overlap. A second, independent adversary, running the function instead of reasoning about it, found the counterexample immediately. Certification by one reviewer is an argument. Reproduction by an independent one is a fact. When you can afford only one, prefer the layer that runs the code.

Can a merge gate itself be a review layer?

The last defect was not in the code at all, and the CI gate surfaced it: my new test used credential-shaped fixtures (values with an AKIA prefix, the AWS access-key-ID shape), and a secret scanner flagged them and transitively blocked the merge, even though that scanner was not one of the branch's required status checks. A second scanner, run on the identical fixtures, passed. The gate disagreed with itself.

The trap is one layer of indirection. The scanner was not in the branch-protection required list, so it looked advisory. But a separate fail-closed meta-check that was required carried its own internal list of checks it waits on, and the scanner was in it. So a "non-required" check gated the merge through a required one. Reading only the branch-protection settings would tell you the merge was clear. It was not.

Two portable lessons fall out. First, never hardcode credential-shaped fixtures; use obviously-fake, low-entropy placeholders that still exercise the code path, because a scanner cannot tell your test data from a live leak and should not have to. Second, when a merge is blocked and the required checks look green, read what your meta-gates actually wait on, not just the protection rules. The scanner also scans full branch history, so neutralizing the fixtures at the tip does not clear a hit an earlier commit introduced; a squash-merge that collapses to the net diff does.

Why don't the layers just catch the same things?

Across the arc, four layers caught five distinct defects with no overlap in the catches (the merge gate alone surfaced two, the credential-shaped fixtures and the transitive block), because each layer senses a different modality. Live recon senses runtime state. The pre-PR fleet senses the diff with fresh adversarial priors. The post-push pass senses by an independent model that runs the code. The CI gate senses policy and scanner state. Independent modalities have independent blind spots, which is the only reason stacking them pays off.

Four review layers, each sensing a different modality. Stacking identical reviewers shares blind spots; independent ones do not.

Review layer	Sensing modality	The defect it caught
Live recon	runtime state on the real host	wrong container selected, false CRITICAL nightly
Pre-PR reviewer fleet	fresh adversarial read of the diff	glob-unquoted redaction needle
Post-push pass	an independent model that runs the code	sequential-substitution partial-secret leak
CI merge gate	policy and scanner state	credential-shaped fixtures, transitive merge block

Stacking identical reviewers does not work. Three passes from the same model with the same framing tend to miss the same things, because they share priors. The gain comes from orthogonality: a runtime probe sees what a reader cannot, a code-runner sees what a reasoner cannot, a scanner sees what a designer forgot. The arc is a clean natural experiment for this, because one layer certified the precise property another layer broke. If the two had shared a modality, they would have shared the blind spot, and the leak would have shipped.

The uncomfortable corollary for AI-written code: the model that wrote the code, the model that reviews the diff, and the model that reasons about correctness are correlated failure sources when they are the same model reasoning the same way. Independence is not a nice-to-have on top of review. Independence is the review.

How do you build this without drowning in process?

You do not need a 20-layer gauntlet on every one-line change; you need a few genuinely independent layers on anything security-adjacent, and a smaller set elsewhere. In practice, for agent-written infrastructure code, four layers cover the modalities that matter, and each is cheap once wired.

Concretely:

Run it against the real environment before you trust the diff. One live execution surfaces the wrong-container, wrong-endpoint, and "works-on-the-dev-box" class that no reader catches. Recon is a layer.
Fan out a small pre-PR reviewer set with distinct framings, not one reviewer three times. Adversarial, correctness, and a security pass on the named bug classes. The union beats the average.
Add one post-push pass by a different model that runs the code, not just reads it. This is the layer that catches what your best reviewer confidently blessed. Weight reproduction over certification.
Treat the CI gate as a reviewer and read its real dependency graph, including what your fail-closed meta-checks wait on. And keep credential-shaped data out of tests entirely.

The through-line is not "more review." It is independent review. Five defects in one 20-line function, each invisible to every layer but one, is the normal shape of software when a fast writer meets a single slow reader. That is not a verdict on the agent; it is arithmetic. Give the writer four orthogonal readers instead, and the blind spots stop lining up.

Building a Governed, Double-Send-Safe Delivery Pipeline for Agent Outputs

Fri, 03 Jul 2026 07:00:00 +0000

I built a multi-agent system to run a small business. Agents drafted work, and some of that work left the building: emails to real people, exported documents, delivered artifacts. I later retired the business on market grounds, but the delivery pipeline is the piece I would rebuild without changing a line. It answers a question most agent demos skip: how do you let an autonomous system send things on a person's behalf without ever double-sending, misfiring, or shipping something nobody approved?

This guide is the pattern, generalized and stripped of the business it ran. It is for anyone whose agent sends consequential output: customer email, outbound messages, exported records, anything where "sent twice" or "sent without sign-off" is a real cost rather than a cosmetic bug.

The one-line version: treat every outbound action as a durable, leased, fail-closed transaction with a human approval gate and an evidence receipt, and make the crash-recovery path refuse to re-send by default. The rest of this guide is that sentence, unpacked.

Why does an agent that sends email send it twice?

The double-send is not a rare edge case; it is the default behavior of a naive queue the first time a worker dies mid-send. A worker claims a send task, hands the message to the mail server, and then is killed (out of memory, a deploy, a reboot) before it writes "done." The task still looks unfinished, so the next worker picks it up and sends again. The customer gets two emails and you get a support ticket.

Every safe-delivery decision below exists to close that window. The failure is structural, not bad luck: the moment of maximum danger is the gap between "the side effect happened in the outside world" and "the system recorded that it happened." A crash in that gap is when a retry becomes a duplicate. You cannot make the gap zero, so you design for a crash inside it.

What does the governed delivery pipeline look like?

The pipeline composes into one spine, and the rule is that no consequential send skips the approval, single-send, and attest gates. The score and review stages decide which items a human reads first rather than gating every send. Six stages, each durable before the next begins:

Produce. The agent generates the artifact in full (for email, a complete RFC 5322 message) and stages it. Nothing is sent inline at generation time.
Persist. Write the intent (the staged artifact plus a queue row and a receipt) to durable storage before anything leaves the process. If you crash now, the intent survives and is inspectable.
Score. Attach a confidence score to the output. This is the input to review routing, not a gate by itself.
Review. Route low-confidence items to a human review queue. High-confidence items still do not bypass the next stage.
Approve. A fail-closed approval gate. The release is blocked by default and proceeds only on an explicit, recorded authorization.
Send and attest. Send exactly once under a lease, then reconcile and write an evidence receipt describing what happened.

The important property is that steps 2 through 6 are separate durable transitions, not one function call. Each stage can crash and resume without losing or duplicating work, because the state that matters lives in the store, not in a worker's memory.

Six durable stages; each arrow is its own transaction, so a crash between stages never loses or duplicates work.

How do you make the send idempotent when a worker dies?

This is the stage most teams get wrong, and the fix is a rule, not a library. Use row-level leasing so exactly one worker owns a task at a time, and make external-send actions non-requeueable on lease expiry. In Postgres the leasing primitive is a claim under SELECT ... FOR UPDATE SKIP LOCKED: concurrent workers select different rows, never the same one, so two workers cannot both send.

The lease is what makes crash recovery safe, and the reaper is where the real decision lives. A reaper reclaims tasks whose lease expired because the worker died. For ordinary internal work, reclaiming and re-running is correct. For an external send it is a loaded gun: if the worker died after the mail server accepted the message but before it recorded success, re-running sends a duplicate. So the rule is:

An external-send action is non-requeueable on lease expiry. When its lease dies, the reaper strands the task for human reconciliation instead of re-running it.

That is the whole trick. A visible stuck task beats an invisible double-send. You trade an automatic recovery you cannot prove is safe for a manual one a human can resolve in seconds by checking whether the message actually went out. Internal, side-effect-free tasks stay auto-requeueable; only the actions that touch the outside world are stranded. Encode the action type in the task and let the reaper branch on it; do not rely on a human remembering the distinction.

The recovery path branches on action type: internal tasks requeue, external sends strand for a human.

How do you keep the pipeline fail-closed?

Fail-closed means the safe state is the default and every unsafe action requires an explicit, positive signal to proceed. Three defaults carry most of the weight:

Sending is off until switched on. A single delivery-enabled flag gates all outbound sends, defaulting to off. A misconfiguration, a fresh environment, or a half-deployed release sends nothing rather than sending wrong.
The sender identity is checked, not assumed. At the point of send, the from-address is checked against a verified sending domain, and a transport guard refuses to send if encrypted transport is expected but the connection would fall back to an unencrypted or wrong port. These checks live in send code, not in a config comment.
Nothing consequential ships without a recorded approval, and everything that ships leaves a receipt. The approval gate is the choke point; the receipt is the audit trail. A stranded, unreconciled send is deliberately left without a success receipt, because its outcome is genuinely unknown until a human checks.

A short operational note, because it cost me. A swallowed import error once left a preflight check silently broken for about 65 days; a reboot finally surfaced it, and the whole time the failure had been read as a benign "dependency unavailable" state. A suppressed error that reads as a normal state is its own failure mode. Fail-closed only helps if the closed state is loud.

When do you actually need this, and when is it overkill?

Match the machinery to the blast radius of a wrong send. For low-stakes internal output (a draft in a channel, a log line, a suggestion a human will obviously see and fix), this full pipeline is overkill and you should not build it. The cost of the ceremony exceeds the cost of the mistake.

You need it the moment a wrong or duplicate send has real cost: customer-facing email, anything touching money, anything with a compliance or legal record, anything a recipient acts on. For those, the pipeline is the floor, not the gold-plating. The tell is simple: if "sent twice" or "sent without sign-off" would generate a support ticket, a refund, or a liability, you are in scope.

The pattern outlived the business I built it for, which is the honest reason it is here rather than in a product. The decision to retire that system, and the principle that a clean architecture is a reason to document the pattern rather than keep funding the business, is its own argument, made in the companion essay Architecture Fit Is Not Business Justification. Build the pipeline when the sends are consequential. Skip it when they are not. And whatever you build, make the reaper refuse to re-send by default.

Architecture Fit Is Not Business Justification

Fri, 03 Jul 2026 07:00:00 +0000

I built a fail-closed, governed multi-agent system to run a first business, and then I retired it. Not because it broke. Because I re-tested the two businesses it was built for against the 2026 market and neither cleared the bar. The architecture being clean was a reason to document the pattern, never a reason to keep the business breathing. Architecture fit is not business justification.

The system was worth building, which is exactly why the temptation to keep it was strong. Nothing consequential shipped without passing a fail-closed policy gate and, where it mattered, a human. Every outbound artifact ran one spine: produce it, persist the intent before anything left, score the confidence and route the low-confidence work to a person, gate the release closed by default, send it once under a lease, then reconcile and write a receipt. The detail I am proudest of is the reaper. When a worker dies after handing an email to the mail server but before recording success, a naive reaper re-queues the task and the customer gets it twice. Mine treats an external send as non-requeueable on lease expiry: it strands the task for a human to reconcile rather than risk a duplicate. A visible stuck task beats an invisible double-send.

None of that saves a business the market has closed. One lane, an agent-run paid-media practice, was being absorbed by the ad platforms themselves; the best-funded pure-play in that space had already pivoted to a human shop and then shut down. The other, a governed compliance-adjacent service, had real demand and real budget, but sat between billion-dollar incumbents bundling the feature for free and headcount shops owning the done-for-you tier. Clean governance changed neither verdict. So I re-tested the thesis before spending more, and when it failed, I stopped.

Killing your own working system on evidence is a competence, not a failure. The salvage is the pattern, never the product. The produce, review, approve, send, attest spine and the double-send-safe reaper outlive the business they were built for, and they map onto every agent that now acts on a person's behalf. A control is only worth what it does to the live target. A business is only worth keeping when the market it serves clears its own test. When it does not, document the architecture and walk.

The Alibi of a Green Gate

Mon, 29 Jun 2026 07:00:00 +0000

A control is what it does to the live target, not what it is declared to be. We build the gate, the reaper, the required check, the backup. We arm it, it runs, it exits zero, the dashboard is green. Then the leak it was meant to kill is still leaking, the bad merge it was meant to refuse already landed, the stale image it was meant to replace is still serving months-old code. Present, configured, armed, inert. None of those four facts is the one that matters.

The failure has one shape, and it is structural. The control's selection model, its liveness model, or its notion of "done" does not match the world it governs, so it sweeps an empty set, or whitelists by naming the bad states it knows and silently passing the ones it does not, or inspects the merged PR instead of the running artifact. Nothing throws. A no-op and a success produce the identical green. The green proves the control ran; it says nothing about whether it bit.

That is the whole trap: "armed and error-free" reads exactly like "remediating," and the reading is free while the proof costs a probe. So pay for the probe. The question is never "did it run." It is "did anything in the live world change because it fired": did the leak shrink, did the merge get refused, did the served bytes turn over. A control that cannot point at an effect on its target has not earned the word "working." Trust the effect, never the exit code. A green gate over an inert control is theater. Prove it bit.

Why 88% of Real LLM Schemas Get Rejected (and the `additionalProperties` Trap Behind It)

Sun, 28 Jun 2026 07:00:00 +0000

I shipped a small tool that lints JSON Schemas against each LLM provider's structured-output constraints, then did the obvious thing: I pointed it at 50 real, public schemas, the kind developers actually feed to these APIs, to check whether the problem I'd built for was real or imagined.

It was real. 44 of the 50 (88%) would be rejected by at least one major provider. Only 3 passed all five. And the single biggest cause is one field almost nobody sets by hand.

This guide walks through what breaks, why it's invisible until production, the one-line shape of the fix, and how to make it a CI check instead of a runtime surprise. Every number here comes from a reproducible run you can repeat on your own schemas (link at the end).

Who this is for: anyone shipping structured outputs, tool calls, or response_format to more than one LLM provider, directly or through a framework like Instructor, LangChain, or an MCP server.

1. The experiment

The corpus was 50 schemas, each traceable to a public source:

15 from the OpenAI and Anthropic cookbooks and SDKs.
17 from popular agent frameworks, Instructor, LangChain, Pydantic-AI, CrewAI, Guidance, LlamaIndex.
18 tool inputSchemas from official Model Context Protocol servers.

Each was linted against OpenAI, Anthropic, Gemini, Mistral, and Cohere. The result:

Provider	Would reject	Notes
OpenAI	88%	Strict Structured Outputs, firm, documented
Mistral	88%	Conservative rule (see the honesty note)
Cohere	26%	Needs ≥1 required field per object
Anthropic	12%	Rejects certain validation keywords
Gemini	0% errors	But 26% warn (`$ref` recursion, `anyOf`)

The portability story is the real headline: only 3 of 50 schemas were clean across all five providers. A schema isn't "valid" or "invalid", it's valid for a specific provider, and the surfaces don't agree.

2. The dominant failure: OpenAI strict Structured Outputs

OpenAI's strict mode is the firmest, best-documented constraint surface, and the one most schemas violate. It enforces two rules that ordinary JSON Schema does not:

Every object must set additionalProperties: false.
Every property must be listed in required. (To make a field "optional," you make it nullable, ["string", "null"], and keep it required.)

Plus a few smaller ones, including: no default, and a restricted keyword set.

Here's a real MCP tool schema, get_channel_history, about as simple as schemas get:

{
  "type": "object",
  "properties": {
    "channel_id": { "type": "string", "description": "The ID of the channel" },
    "limit": { "type": "number", "description": "Number of messages", "default": 10 }
  },
  "required": ["channel_id"]
}

Two properties. It passes Anthropic, Gemini, and Cohere. Against OpenAI strict mode it fails three ways at once:

no additionalProperties: false on the object,
limit is optional (not in required),
limit carries a default.

This isn't a contrived edge case, it's the default shape of a hand-written tool schema, and the default output of most Pydantic models (additionalProperties simply isn't emitted, and optional fields stay out of required). Across the 50-schema corpus, additionalProperties alone accounted for 134 flags, more than every other keyword combined.

3. Why it's invisible until production

The constraints are documented, so why does everyone hit them? Because the failure mode is opaque. When a provider rejects a schema, the API returns the equivalent of "invalid schema", a 400, without telling you which constraint, at which path, failed. So the loop becomes: ship → 400 in production (or in a flaky eval) → bisect the schema by hand → discover it was a missing additionalProperties three levels deep in a nested object.

Multiply that by five providers, each with a different surface, and "just hand-port the schema" becomes a recurring tax instead of a one-time fix.

4. The shape of the fix

For the Slack example, OpenAI-strict-valid looks like:

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "channel_id": { "type": "string", "description": "The ID of the channel" },
    "limit": { "type": ["number", "null"], "description": "Number of messages" }
  },
  "required": ["channel_id", "limit"]
}

additionalProperties: false added, limit made nullable and moved into required, default dropped. The semantics are preserved (a caller can still pass null for limit), but now it satisfies strict mode.

The mechanical part is easy once you know which rule fired and where. The hard part is knowing, before the 400.

5. The cross-provider trap

OpenAI is the loudest case, but the deeper problem is that the five surfaces disagree, so fixing for one can break another:

Anthropic rejects a set of validation keywords in tool input schemas, format, minLength, pattern, minimum/maximum, that OpenAI and others accept. A schema you tightened for OpenAI can trip Anthropic.
Gemini has no anyOf/union support in the same way and chokes on self-referential $ref (recursion), a common shape when a Pydantic model references itself. In the corpus these showed up as warnings, not hard errors, but they're real runtime risks.
Cohere wants at least one required field on every object.
Mistral's structured-output examples set additionalProperties: false everywhere, like OpenAI.

This is why only 3 of 50 schemas passed all five. There is no single "correct" schema, there's a per-provider constraint surface, and a schema that ships to multiple providers has to satisfy the intersection.

6. Catch it in CI, not in prod

The fix for an opaque, multi-surface, easy-to-regress constraint is the same fix we use for every other class of "works on my machine" bug: a static check that fails the build.

That's the entire premise of schemafit, the tool I ran this study with. It encodes each provider's documented constraint surface as a versioned rule pack and lints your schema statically: exact JSON-Pointer path, the keyword, and why, with a non-zero exit code so CI fails the PR instead of prod. It makes no model calls, needs no API key, and has zero runtime dependencies.

pip install schemafit
schemafit lint my-schema.json --provider openai,anthropic,gemini,mistral,cohere

It ships a GitHub Action and a pre-commit hook, and emits SARIF for GitHub code-scanning. But the tool is incidental to the lesson: treat provider schema constraints as a CI concern, the way you treat types or lint. Whether you adopt a linter, a test, or a checklist, the move is to surface the 400 before it reaches a provider.

7. Honesty notes (because the numbers will get scrutinized)

OpenAI's 88% is the firm case. Strict Structured Outputs documents the additionalProperties: false and all-required requirements; the schemas that passed OpenAI genuinely carried both. That number is solid.
Mistral's 88% uses a more conservative rule. Mistral's docs show additionalProperties: false in their structured-output examples but don't prose-mandate it, so read Mistral as "likely-reject," not "documented-reject."
Gemini's hits were warnings, not errors, $ref recursion and anyOf are real risks, but I'm not claiming a guaranteed 400.
The corpus is provider-mixed by design, cookbook examples skew already-strict; framework and MCP schemas skew permissive, so 88% is a floor, not a cherry-pick.

The full per-provider breakdown, the provenance for all 50 schemas (every source URL), and a script to reproduce the run on your own schema directory are in the repo's benchmarks/.

The takeaway

LLM schemas aren't portable, and the providers won't tell you why a schema failed, only that it did. The constraint surfaces are documented but scattered, opaque at the API boundary, and easy to regress when you fix for one provider and break another. The 88% number isn't an indictment of anyone's schemas; it's evidence that this belongs in CI, next to your other static checks, not in a production traceback at 2am.

Run it on your own schemas and see where you land. If you hit a rejection a linter doesn't catch, or a rule that's wrong for your setup, that's exactly the feedback worth sharing.

Self-Correcting Agents: Learning the Loop the Hard Way

Sat, 27 Jun 2026 07:00:00 +0000

I ran a multi-agent research agent over a hard question and it came back with a clean, confident verdict: "All 25 claims refuted by adversarial verification. Research inconclusive."

Every one of those 25 claims was true. Several cited real, recent papers I could pull up in another tab. Nothing had actually been checked.

The verifier hadn't disagreed with the research. It had crashed, every verification call hit transient API rate-limiting and returned nothing, and the loop scored "returned nothing" as "refuted." A fail-closed gate that cannot tell refuted by evidence apart from never adjudicated will lie to you with total confidence, and dress the lie up as diligence.

This guide is about that failure mode: why it happens, why it's the default outcome unless you design against it, how to spot it in your own loops, the gate that prevents it, and how to recover a run that has already produced a false "everything is false."

Who this is for: anyone building agent loops that check their own work, research agents, code agents with a review pass, anything with a "generate → verify → keep or discard" cycle. You don't need a specific framework. You need to take one idea seriously: silence is not a verdict.

1. The anatomy of a self-checking loop

Almost every serious agent loop has the same three roles, whatever you call them. The reasoning-frontiers survey of LLM reasoning fixes the vocabulary (arXiv:2504.09037):

A Reasoner generates candidate answers (the policy).
One or more Verifiers evaluate them (the judge / reward signal).
A Refiner revises based on verifier feedback.

All three can be the same model wearing different hats, that's exactly what "self-refinement" is. The research agent that lied to me was a fan-out version of this:

scope ──► search ──► fetch ──► extract claims ──► VERIFY (3-vote panel per claim) ──► synthesize
                                                      │
                                                      └─ each claim needs a quorum of votes to survive

The verify stage is the load-bearing one. It's where the loop decides what's true enough to keep. And it's the stage everyone gets wrong, because it's the stage where the failure is invisible: a broken Reasoner produces obvious garbage, but a broken Verifier produces clean, plausible, wrong verdicts.

2. The bug: a two-state gate in a three-state world

Here's the gate that shipped in the loop that failed me. Each claim got three independent verifier votes, tallied as notRefuted, refuted. The survive rule was reasonable on its face:

A claim survives if it has at least 2 valid votes and fewer than 2 refutations.

Read that rule carefully. It quietly assumes votes exist. When the verify phase crashed under rate-limiting, every vote came back empty. Each claim scored 0-0: zero refutations, but also zero valid votes. The survive rule said "fewer than 2 refutations? yes. At least 2 valid votes? no.", so the claim didn't survive.

So far, so correct. Refusing to pass an unverified claim is good design. The bug was one layer up, in the reporting: the claims that didn't survive were dumped into a bucket literally named refuted[], and the summary rendered that bucket as "all 25 claims refuted, research inconclusive."

The gate had collapsed a three-state world into two states:

A verifier vote has three outcomes, not two. Folding ABSTAINED into REFUTED is how a crash becomes a confident 'no.'

The world has three outcomes, and you must keep them distinct:

Outcome	Meaning	What it tells you
CONFIRMED	quorum of valid votes, below the refute threshold	keep it
REFUTED	quorum of valid votes, at/above the refute threshold	drop it, the evidence says no
ABSTAINED	below quorum of valid votes (crash, timeout, null, rate-limit)	you don't know yet, re-run, escalate, or surface as pending

The one rule this whole guide rests on: an absence of votes is not a vote. Refuted is a verdict the evidence reached; abstained is a verdict the evidence never reached. A gate that conflates them converts every transient outage into a wall of false negatives, and because it fails closed, it looks responsible while doing it.

3. Why this is the default, not an edge case

You might think this is an exotic bug. It isn't, it's what you get unless you specifically prevent it, for three compounding reasons.

Fail-closed is the right instinct, and that's the trap. You should refuse to pass a claim you couldn't verify. Every safety-conscious engineer builds the gate to deny on uncertainty. The problem is that "deny" and "refute" land in the same bucket unless you force them apart. The safer your instinct, the more likely you are to mislabel silence as rejection.

Verifiers fail in correlated bursts. A single flaky vote is noise you'd shrug off. But verifier calls share infrastructure, the same API, the same rate limiter, the same network. When one abstains from a transient limit, they all do, at once. So you don't get one mislabeled claim; you get the whole run mislabeled in a single stroke. The blast radius is the entire output.

The summary layer launders the error. The gate's internal state might still be technically recoverable (0-0 is visibly different from 0-2). But by the time it reaches a one-line summary, "research inconclusive", the distinction is gone. A human reading the summary has no way to tell a crash from a conclusion. The error becomes load-bearing for every downstream decision.

This is a specific, nasty instance of a general truth the research keeps surfacing: *agents are far better at producing a good answer than at recognizing one.* The verification step is the weakest link in the loop, so it deserves the most defensive design, not the least.

4. The research: verification is the bottleneck, not generation

Step back from the bug and look at why the verify stage is where loops live or die.

The verification gap. A 2026 benchmark of general LLM agents put a name to the failure (arXiv:2602.18998): when you sample several candidate answers, the right one is often already in your set, your pass@K is high, because the model generated it, but the model can't reliably select it, so the accuracy you actually realize lags far behind the answer you already have. Generation outruns judgment. This is the structural reason "just let the agent double-check itself" underdelivers, and the reason verifier quality dominates loop quality.

The self-correction blind spot. It gets sharper. Self-Correction Bench measured an average 64.5% blind-spot rate across 14 models (arXiv:2507.02778): models readily fix an error when it's pointed out in the prompt, but systematically fail to fix the same error in their own prior output. The likely cause is mundane, training data is overwhelmingly composed of clean, correct demonstrations, not error-then-correction traces, so the "notice and revise my own mistake" behavior is under-trained. The striking part: simply appending the word "Wait" cut blind spots by 89.3%. The capability is there; it just doesn't fire on its own.

Put those two findings together and the design conclusion is blunt: a loop that relies on a single model to silently self-judge is building on the part of the system that is measurably worst at the job. You cannot fix that with more self-reflection. You fix it with more, and more independent, verification, which is exactly why a verifier that quietly drops to zero is so destructive. It removes the one thing that was compensating for the model's blind spot, and it does so invisibly.

Verification is its own scaling axis. The frontier response to the verification gap is to scale the number of verifiers, not the cleverness of one. Multi-Agent Verification (arXiv:2502.20379) uses off-the-shelf models as independent "Aspect Verifiers" that vote, no training required, and shows weak-to-strong generalization: a panel of weaker verifiers can improve even a stronger generator. Rubric-grounded test-time verification (DeepVerifier, arXiv:2601.15808) extends the same idea to research agents, plug-and-play with no fine-tuning. The whole point of these systems is to make the verdict robust. A panel that silently abstains en masse is the precise opposite, it's a quorum that quietly dropped below quorum without telling anyone.

5. The fix: design the gate around three states

The fix is not subtle once you've named the problem. Make ABSTAINED a first-class state and never let it masquerade as either pass or fail.

Three rules implement it.

*Rule 1, classify on valid votes, not on net score. A vote only counts toward a verdict if the verifier actually ran and returned a parseable judgment. Before you tally notRefuted vs refuted, count how many votes are valid at all*. If valid votes are below your quorum, the outcome is ABSTAINED, full stop, you never even reach the refute comparison.

for each claim:
    valid   = votes that actually ran and parsed
    refuted = valid votes that say "refuted"
    if len(valid) < QUORUM:                -> ABSTAINED   (not adjudicated)
    elif len(refuted) >= REFUTE_THRESHOLD: -> REFUTED     (evidence says no)
    else:                                  -> CONFIRMED    (keep)

For the 3-vote panel from §2, QUORUM = 2 and REFUTE_THRESHOLD = 2, the very same thresholds as the broken gate. Nothing about the numbers changed; only the order of the checks and the keying on valid-vote count did.

The single change from the broken gate is the first branch runs first and is keyed on valid-vote count, not on the refutation count. A 0-0 claim can never be reported as refuted because it exits at ABSTAINED before the refutation branch is ever evaluated.

Rule 2, an error, timeout, or null is an abstention, and abstentions get retried, never counted clean. A verifier invocation that throws, times out, or returns nothing is a PENDING abstention, re-run it up to N times with backoff. Critically, an abstention is never silently treated as a pass or a fail. (This is the same discipline a good code-review gate needs: a reviewer subagent that errors out hasn't approved the diff, treating "the review crashed" as "the review is clean" is the same bug wearing a different shirt.)

The retry loop sits between 'vote arrived' and 'verdict counted.' Exhausted retries surface as PENDING, never as pass or refute.

Rule 3, never transcribe "inconclusive" as a finding without checking whether the verifiers ran. This is the reporting-layer rule, and it's the one that actually saved me. Before a summary says "all refuted" or "inconclusive," it must inspect the shape of the votes that produced that verdict. If the run is dominated by 0-0 abstentions, the honest summary is "verification did not complete", an operational failure to retry, not "the claims are false," a substantive finding. A crashed adjudicator's silence must never be rendered as a conclusion.

A compact way to enforce Rule 3 is a tiny, report-only reclassifier that you can run over any saved run output:

reclassify(run_output):
    for each claim in run_output.refuted_bucket:
        valid = parse_vote_tally(claim)
        if valid < QUORUM:  ->  ABSTAINED   (recoverable; the run never judged it)
        else:               ->  REFUTED     (a real verdict)
    if ABSTAINED dominates:  print "ABSTENTION-DOMINATED, the run did not refute anything"

(One naming note so the labels don't trip you: ABSTAINED is the terminal "not adjudicated" state, the same one a run's output later shows as "unverified"; PENDING is just its mid-retry sub-state. One concept, named once.) It changes nothing and runs nothing, it just re-reads the tallies and tells you whether your "all refuted" was a verdict or a crash. That single check is the difference between trusting a false negative and catching it.

6. Detecting it in a loop you already have

You can audit an existing loop for this failure without rebuilding it. The tells:

Symptom	What it usually means
A verify/review stage reports everything failed at once	correlated abstention (shared rate limit / outage), not a real mass rejection
Verdicts flip between runs with no input change	the gate is scoring transient failures as outcomes
The "failed" bucket has no per-item evidence (no refuting quote, no reason)	those items were never adjudicated, there's nothing to cite because nothing ran
A summary says "inconclusive" but the underlying items cite real, checkable sources	the loop discarded true results; verification, not the research, is what failed
Tightening a rate limit or shrinking a batch makes the "failure rate" drop	you're measuring infrastructure, not correctness

The deepest tell is the absence of evidence attached to a negative verdict. A genuine refutation can point at why, a contradicting source, a failing assertion, a counterexample. An abstention can't, because nothing happened. If your "no" can't show its work, it isn't a no.

7. Recovery: resume, don't re-run

Say it already happened, a run produced a false "everything refuted." The instinct is to run the whole expensive pipeline again from scratch, and in a real fan-out (mine spanned several stages and a couple of dozen sources), "from scratch" is a lot of wasted minutes and tokens. Don't. Most of that run is fine and recoverable. Recover in three steps, cheapest first.

Recovery is cheapest-first: reclassify (seconds) → resume from cache (only the crashed calls re-run) → sequential top-up for the tail.

Step 1, reclassify (seconds, changes nothing). Run the report-only reclassifier from §5 over the saved output. It splits the "refuted" pile into genuinely refuted (a real quorum said no) versus abstained (below quorum, recoverable). If it comes back abstention-dominated, you now know the research itself is intact; only the verification failed. You've turned a false catastrophe into a known, bounded gap.

Step 2, resume from the cached prefix; don't restart. A good loop harness caches completed stages. The scope, the search, the fetch, and every verifier vote that actually succeeded are still valid, re-running them wastes time and burns the same rate limit that caused the failure. Resume the run so that only the abstained verifier calls re-execute. You pay for the gap, not the whole pipeline.

Step 3, top up the tail by hand, gently. For the handful of items still unverified after the resume, verify them sequentially, one source at a time, at a calm cadence. The shared rate limiter that caused the crash eases over time; a gentle, serial top-up slips under it where a parallel burst re-trips it. This is slow on purpose. Slow-and-complete beats fast-and-false.

A note on "future-dated" citations: when you re-verify, resist the urge to auto-reject a source just because it looks improbable, a paper with a publication date a few months out, say. In my run, several "suspicious" citations resolved to entirely real papers on inspection. Existence-check before you refute. That, too, is the difference between a verdict and a guess.

8. The broader principle: spend compute on generation and external verification

Zoom all the way out. The abstention bug is a sharp instance of a general lesson about where inference compute pays off in loops, and the evidence is increasingly clear that the popular instinct is backwards.

Brute repetition is the weak lever; verification is the binding constraint. Spending more compute by sampling several independent trajectories and selecting among them can help, the systematic study of test-time scaling for agents finds that scaling test-time compute, diversifying rollouts, and list-wise verification all move the needle (arXiv:2506.12928). But the payoff is far from automatic: a 2026 benchmark of general LLM agents found that neither parallel sampling nor longer sequential reasoning reliably improved performance, sequential scaling hit an "effective context ceiling," and parallel scaling was capped by the verification gap itself (arXiv:2602.18998). Read that twice: the thing throttling parallel sampling is the same selection problem this whole guide is about. The right answer is often already in your samples; the agent just can't pick it, and letting one agent simply "think longer" is the weakest lever of all.

Reflect selectively, not constantly. When you do refine, trigger it only when a step's score falls below a threshold, rather than reflecting at every step, knowing when to reflect beats reflecting often (arXiv:2506.12928). And self-verify-then-correct loops (SETS, arXiv:2501.19306) keep improving with compute where plain best-of-N plus majority vote saturates. The throughline: structured selectivity and stronger verification scale; brute repetition doesn't.

Now connect that back to the bug. If parallel generation puts the right answer in your candidate set, and the verification gap means selecting it is the hard part, then your verifier is the highest-leverage component in the entire loop. Which is exactly why a verifier that can silently fall to zero, and report that zero as a confident "no", is the most expensive bug you can ship. It doesn't just add noise; it removes the one mechanism that was carrying the loop, and it hides the removal.

The design conclusion, stated as a single sentence: spend your inference budget on diverse generation plus stronger, independent, external verification, and build that verification so that when it fails, it says "I didn't check," never "the answer is no."

9. Checklist: an honest verification gate

Use this when you build or audit a self-checking loop.

[ ] Three states, always. Every verdict is CONFIRMED, REFUTED, or ABSTAINED. There is no fourth bucket, and ABSTAINED is never quietly merged into either of the others.
[ ] *Classify on valid votes first. Count votes that actually ran and parsed before* you compare refutations. Below quorum of valid votes ⇒ ABSTAINED, and you exit before the refute branch.
[ ] Errors/timeouts/nulls are abstentions. A verifier that throws, times out, or returns nothing is PENDING, retry up to N with backoff. It is never counted as clean and never counted as refuted.
[ ] A negative verdict must show evidence. A real REFUTED cites why (a contradicting source, a failing check). If a "no" has no attached evidence, treat it as an abstention until proven otherwise.
[ ] Guard the summary layer. Before any "inconclusive"/"all refuted" summary ships, inspect the vote shape. Abstention-dominated ⇒ report "verification did not complete," not "the claims are false."
[ ] Keep a report-only reclassifier. A tiny tool that re-reads a saved run's tallies and splits refuted from unverified turns a silent false-negative into a one-command catch.
[ ] Recover by resume, not restart. Cache completed stages and successful votes; re-run only the abstained calls. Top up the tail sequentially and gently so you don't re-trip the limiter.
[ ] Watch for correlated failure. If a whole batch "fails" at once, suspect shared infrastructure (rate limit, outage) before you believe a mass rejection.

The discipline behind every line is one sentence worth memorizing: a verifier's silence is missing data, not a "no." Build your loop so it can tell the difference, and it will fail honestly, which is the only kind of failure you can recover from.

Further reading: the systematic treatment of test-time scaling for agents (arXiv:2506.12928) and the 2026 benchmark of general agents that finds brute test-time scaling underdelivers, naming the context ceiling and the verification gap (arXiv:2602.18998); Multi-Agent Verification / Aspect Verifiers (arXiv:2502.20379) and rubric-guided test-time verification (arXiv:2601.15808) for verification as its own scaling axis; Self-Correction Bench for the 64.5% blind spot and the "Wait" activation (arXiv:2507.02778); SETS for self-verify-and-correct loops (arXiv:2501.19306); and the reasoning-frontiers survey for the Reasoner/Verifier/Refiner vocabulary (arXiv:2504.09037).

Self-Hosting Many Apps on One Box: A Beginner's Guide to the Two-Plane Architecture

Thu, 25 Jun 2026 07:00:00 +0000

If you've ever tried to run your own apps, a dashboard, wiki, side-project API, or metrics stack, on a single machine you control, you've probably hit the wall everyone hits: self-hosting gets messy fast. A port opened here, a hand-written nginx file there, a forgotten admin panel, a leaked origin IP, and suddenly your "one box" is an attack surface you can't reason about.

This guide gives you a clean, defensible architecture instead. By the end, the recommended path gives you *one box, one reverse proxy, zero public inbound ports, and a private mesh for everything you administer, and a mental model that scales from one app to dozens without the security story getting worse. ("Zero public inbound ports" is the precise claim: Traefik still listens inside* the box; nothing is published to the public internet.)

Who this is for: a capable beginner comfortable with Docker, a terminal, and editing YAML. You don't need to be a network engineer. You do need to be willing to think in terms of planes instead of ports.

1. Why self-host (and what you're actually signing up for)

The payoff is real: full control, no per-seat SaaS bills, your data on your own box, and the ability to run many apps on one machine. You also learn how the internet actually works, DNS, TLS, proxies, VPNs, which pays dividends forever.

The honest cost: you are now the ops team. Uptime, TLS renewal, backups, and security are yours. This guide minimizes that burden; it doesn't erase it.

The naive approach, and why it fails: you open a firewall port per app, hand-write a vhost per app, and point public DNS straight at your home or VPS IP. This sprawls (every app is a config island), leaks your origin IP (now you're a direct DDoS and scan target), and gives an attacker one open port per service to probe. Three apps, three ports, three things that can be misconfigured into a breach.

Naive self-hosting exposes many doors; the two-plane model exposes one controlled front door and keeps admin access private.

What we build instead: a single reverse proxy as the only public front door, a tunnel so you open zero inbound ports, and a private mesh VPN for every surface only you should touch.

2. The mental model: two planes

Here is the single idea the whole guide rests on. Every service you run answers one question:

Does the public need to reach this?

Yes → it lives on the PUBLIC INGRESS plane. (Your apps, your APIs, anything a visitor uses.)
No → it lives on the PRIVATE ADMIN plane. (Dashboards, metrics, databases, secrets stores, SSH, everything you touch as the operator.)

These two planes have separate entry points, separate failure domains, and separate trust models. That separation is the security story: a misconfigured public route cannot expose an admin tool that has no public listener at all.

The load-bearing rule, stated up front:

Admin/internal services bind only to loopback (127.0.0.1) or the mesh interface, never 0.0.0.0. You defend by not exposing, not by hardening an exposed port.

0.0.0.0 means "every interface, including the public one." If a database only listens on the mesh, there is literally no public port for anyone to scan or brute-force.

One thing to get right from the start: the "mesh interface" is your Tailscale interface, whose IP comes from Tailscale's CGNAT range 100.64.0.0/10 (addresses 100.64.x, 100.127.x), not a private-LAN range like 10.0.0.0/8 or 192.168.0.0/16. Those LAN ranges are a different network entirely (often routable on your physical network), so allowlisting them is a security hole, not the mesh. Everywhere this guide says "mesh range," it means 100.64.0.0/10.

Here's the whole architecture in one diagram:

Public users enter through Cloudflare and Traefik; operators enter through the private Tailscale mesh.

Visitors take the five-layer public path. You take the mesh. The two planes meet only through explicitly declared routes, a public route for a public app, or a mesh-only route for an admin tool. No service gets both by accident.

Known-good default stack, keep every example matching this and you avoid the configuration drift that breaks beginner setups:

Recommended default for this guide
──────────────────────────────────
• cloudflared      → Docker container (same network as Traefik)
• Traefik          → Docker container, the only front door
• Public apps      → Docker containers behind Traefik
• Admin apps       → loopback or Tailscale only, never published
• Public DNS       → Cloudflare proxied hostname → Tunnel
• Public ports     → NONE published to the host's public interface

3. Prerequisites and one-time setup

You'll need:

A Linux box (VPS or home server) with Docker and Docker Compose installed.
A domain you control, with DNS managed by a free Cloudflare account.
Cloudflare Zero Trust enabled (the free tier covers Tunnels and Access).
A Tailscale account with the client installed on both the box and your laptop. One login enrolls a node.
A secrets habit from day one. Decide where secrets live, an env file kept out of git, or a secrets manager, before you wire anything. Never commit a key.

Sanity check before going further: confirm docker run hello-world works, the box has outbound internet, and you can ssh to it over Tailscale. If those three pass, you're ready.

4. Cloudflare Tunnel: reaching the box with zero open ports

The problem it solves. Port-forwarding is the classic way to expose a home server, but it requires an open inbound port (a permanent attack surface), publishes your origin IP, and simply doesn't work behind CGNAT or a dynamic IP.

How it fits. A small daemon, cloudflared, runs on your box and *dials out* to Cloudflare, holding the connection open. Inbound requests ride back down that existing connection. You open nothing** inbound. Your origin IP is never in public DNS. Its only requirement is outbound egress to Cloudflare, which is why it's the recommended default for beginners.

Bootstrap the tunnel (locally-managed path). Before any config file means anything, you have to create the tunnel and its credentials. This sequence assumes a locally-managed tunnel, created from the CLI, configured by your on-box config.yml:

# 1. Authenticate cloudflared to your Cloudflare account (opens a browser).
cloudflared tunnel login

# 2. Create the tunnel. This PRINTS the <tunnel-id> and WRITES the
#    credentials file (~/.cloudflared/<tunnel-id>.json) that config.yml points at.
cloudflared tunnel create my-box

# 3. Create the proxied CNAME for your hostname (see the DNS note below).
cloudflared tunnel route dns my-box app.example.com

# 4A. DEFAULT for this guide, run cloudflared as a Docker container on the
#     SAME network as Traefik (so the name "traefik" resolves). See the
#     cloudflared compose service below (mounts config.yml + <tunnel-id>.json).
cloudflared tunnel run my-box          # optional quick foreground test
# 4B. Host alternative, install as a systemd service. Use this ONLY if your
#     ingress routes to http://127.0.0.1:<port>, never to a Docker name.
sudo cloudflared service install

For the default stack, run cloudflared as a container on the same proxy network as Traefik:

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    command: tunnel --config /etc/cloudflared/config.yml run
    volumes:
      - ./cloudflared/config.yml:/etc/cloudflared/config.yml:ro
      - ./cloudflared/<tunnel-id>.json:/etc/cloudflared/<tunnel-id>.json:ro
    networks: [proxy]
    restart: unless-stopped

networks:
  proxy:
    external: true

Step 2 is what produces the <tunnel-id> and the credentials JSON referenced below, without it, the config.yml points at a file that doesn't exist. The daemon must actually be running, however you run it; this guide assumes the container path.

A generic tunnel ingress config (config.yml):

tunnel: <tunnel-id>
credentials-file: /etc/cloudflared/<tunnel-id>.json   # the file `tunnel create` wrote

ingress:
  # DEFAULT, cloudflared and Traefik are containers on the same Docker network,
  # so the name "traefik" resolves. Cloudflare terminates the browser-facing TLS
  # at its edge, so the tunnel's hop to Traefik's plain :80 entrypoint is fine.
  - hostname: app.example.com
    service: http://traefik:80
  # Catch-all MUST be last (required whenever you define ingress rules).
  - service: http_status:404

The addressing fork, get this right or nothing routes. The service URL depends entirely on where cloudflared runs:
cloudflared as a Docker container (this guide's default): use service: http://traefik:80. The name traefik resolves only because both containers share the Docker network.
cloudflared as a host systemd service (cloudflared service install): a host process cannot resolve Docker service names. Use service: http://127.0.0.1:<port>, and have Traefik publish a host port bound to loopback only, e.g. ports: ["127.0.0.1:8080:80"]. Loopback isn't public, so the zero public inbound ports story holds.
Do not mix the two. A host-installed cloudflared pointing at http://traefik:80 is the single most common way this setup fails, that name resolves only inside Docker. On the host, route to loopback or a real host IP, never a Docker service name.

Where cloudflared runs determines whether the tunnel targets Docker DNS or loopback.

Two things people conflate here:

(a) Encrypting the cloudflared → Traefik hop. Point the ingress at https://traefik:443 and define a websecure (:443) entrypoint (§5). If Traefik serves an internal/self-signed cert, originRequest.noTLSVerify: true makes that work, but it disables origin-cert verification, so this is not "validated" origin TLS.
(b) Cloudflare-validated origin TLS. Use the published-:443-origin variant: publish only 443, present a Cloudflare Origin Certificate (or a publicly-trusted cert), and set Cloudflare SSL/TLS mode to Full (strict), the only mode that actually validates the origin cert.
Through a tunnel the browser always validates Cloudflare's edge cert, never Traefik's, so "browser-validated origin TLS" only exists on the published-origin path. The default tunnel + edge-TLS path needs none of this.

DNS wiring, and who creates the record. For a CLI-created (locally-managed) tunnel, cloudflared tunnel route dns (step 3 above) creates the proxied CNAME for you, already orange-cloud, mapping app.example.com to <tunnel-id>.cfargotunnel.com. Don't also hand-create it, a second record risks a duplicate or a wrong-tunnel-id conflict. Hand-create the proxied CNAME yourself only for dashboard/remotely-managed tunnels or when you're routing manually. Note that cfargotunnel.com records are proxied-only by design, Cloudflare won't let you grey-cloud them, so the orange-cloud "master switch" is a real operator choice on the published-:443-origin path, not on the tunnel path.

The #1 gotcha, locally-managed vs remotely-managed tunnels. Cloudflare now nudges most people toward remotely-managed tunnels (configured in the dashboard); this guide uses the locally-managed (CLI + config.yml) path because config-as-code is easier to version, review, and reason about. The catch: if you created the tunnel in the Cloudflare dashboard (or via the API), the remote config is authoritative and your local config.yml is ignored. Edit ingress in the dashboard/API in that case. And when you update a remotely-managed tunnel via API, you must re-send the entire ingress list, it's a replace, not a merge. Everything else in this section assumes the locally-managed (CLI) path above.

Optional identity gate. Put Cloudflare Access in front of a hostname for SSO/OTP login at the edge, with zero app code. Carve a path-scoped policy with action Bypass (e.g. /healthz) so uptime probes stay green without logging in.

Verify correctly:

A redirect to the Access login page is the healthy gated state, not a failure: browser flows get a 302; API/service-token flows get a 401 or 403. Any of those means "gated," not "broken."
To prove your origin actually routes, curl the bypassed path through real public DNS and expect a real 200, but this only holds if that path is covered by a policy whose action is Bypass (evaluated first, no identity, no logging). An Allow policy still challenges for identity, so it will 302 even on the "bypassed" path.
An empty-body 404 with server: cloudflare means the hostname has no ingress rule and never reached your origin.

5. Traefik: route every public app through one internal front door

The problem it solves. You have one IP and one pair of ports (:80/:443) but want to serve dozens of apps on different subdomains, without exposing a port per app or restarting anything to add a route.

How it fits. One Traefik container listens on internal entrypoints and routes each request to the right backend container by Host (and optional path). In the default tunnel path, browser-facing TLS terminates at Cloudflare's edge and Traefik routes plain HTTP inside the Docker network; in the published-origin or mesh-TLS variants, Traefik also serves HTTPS on websecure. Routes hot-reload.

The #1 mental model: static vs dynamic config.

Static (entrypoints, ports, providers, cert resolvers) is set at startup and needs a restart to change.
Dynamic (routers, services, middlewares, certs) is hot-reloaded, no restart.

Put each setting in the right layer and you'll never fight the proxy.

Static config boots Traefik; dynamic config routes traffic.

Static config (traefik.yml), entrypoints, providers, cert resolvers. Install-time concerns: set once, restart to change. The default tunnel path dials the web (:80) entrypoint; the websecure (:443) entrypoint is here only for the published-origin TLS variant. Whichever entrypoint a route uses must exist, or cloudflared gets connection-refused:

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

providers:
  docker:
    endpoint: "tcp://dockerproxy:2375"   # the read-only socket proxy (see safety rules)
    exposedByDefault: false              # nothing is exposed unless it opts in
    network: proxy
  file:
    filename: /etc/traefik/dynamic.yml
    watch: true

# NOTE: TLS certificates/stores are DYNAMIC config, not static, they live in
# the `tls:` block of dynamic.yml below. Static config is only entrypoints,
# providers, cert resolvers, logging, and the dashboard toggle. (The default
# tunnel path needs no origin cert at all: Cloudflare terminates TLS at its edge
# and the tunnel dials plain :80. The cert only matters for the :443 variant.)

A generic docker-compose.yml for an app declaring its own route via labels:

services:
  my-app:
    image: my-app:latest
    networks: [proxy]            # shared proxy network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.my-app.rule=Host(`app.example.com`)"
      - "traefik.http.routers.my-app.entrypoints=web"   # default tunnel path (edge TLS); use websecure + tls=true only for the published-origin variant
      - "traefik.http.services.my-app.loadbalancer.server.port=8080"

networks:
  proxy:
    external: true

Note traefik.enable=true is explicit, because the static config sets exposedByDefault=false, nothing is exposed unless it opts in. Admin containers carry traefik.enable=false.

The same routes, expressed in a watched dynamic file (dynamic.yml), useful for backends Traefik doesn't manage (raw IP:port, cross-host services):

http:
  routers:
    my-app:
      rule: "Host(`app.example.com`)"
      entrypoints: ["websecure"]
      service: my-app-svc
      priority: 1                  # catch-all: lowest priority
      tls: {}                      # NOT a no-op: "serve over TLS with the default cert"
    my-app-api:
      rule: "Host(`app.example.com`) && PathPrefix(`/api`)"
      entrypoints: ["websecure"]
      service: my-api-svc
      middlewares: ["strip-api"]
      priority: 10                 # higher than the catch-all above
      tls: {}

  services:
    my-app-svc:
      loadBalancer:
        servers:
          - url: "http://my-app:8080"
    my-api-svc:
      loadBalancer:
        servers:
          - url: "http://my-api:9000"

  middlewares:
    strip-api:
      stripPrefix:
        prefixes: ["/api"]         # backend sees a clean path, a frequent 404 fix
    mesh-only:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "100.64.0.0/10"        # Tailscale CGNAT mesh range, NOT 10.x/192.168.x

tls:                               # top-level key, TLS certs are DYNAMIC config
  stores:
    default:
      defaultCertificate:          # served on :443 by default, published-origin variant only
        certFile: /run/secrets/origin.crt   # e.g. a Cloudflare Origin Certificate
        keyFile: /run/secrets/origin.key

A few things that bite beginners:

Empty tls: {} is not a no-op, it means "serve this over TLS with the default cert." Omit it and the router answers plain HTTP. And note which cert "default" is: if you haven't configured a defaultCertificate or an ACME resolver, it's a generated self-signed cert, fine behind Cloudflare with noTLSVerify, but a browser hitting the origin directly will warn.
Priority matters the moment a host has both a catch-all and a specific path. Give the catch-all priority: 1 and the specific path route a higher number, or the catch-all swallows /api.
stripPrefix drops /api before proxying, so the backend sees a clean path. Forgetting it is a classic source of 404s.
ipAllowList restricts an admin route to loopback + the Tailscale mesh range 100.64.0.0/10, but only when the request actually arrives over the mesh (see the caveat below). Do not put 10.0.0.0/8 or 192.168.0.0/16 here: those are private-LAN ranges, not your tailnet, and allowlisting them opens the route to your whole LAN instead of locking it to the mesh.

Critical caveat, ipAllowList matches the IP Traefik actually sees. Behind a Cloudflare Tunnel, the request reaches Traefik from the cloudflared connector (or the Docker bridge), not your laptop's 100.64.0.0/10 tailnet IP. So ipAllowList: 100.64.0.0/10 does nothing on a public, Cloudflare-fronted hostname. It only gates a route reached directly over Tailscale (a tailnet hostname / tsnet entrypoint, §6). Never put a mesh allowlist on a public route and assume it keeps admins-only.

The corollary: the right control depends on how the route is reached.

Route type	The control that actually works
Public hostname via Cloudflare	Cloudflare Access (SSO), app auth, service tokens, WAF rules
Mesh-only hostname / listener	Tailscale ACLs + bind policy + (optionally) Traefik `ipAllowList`
Local-only admin tool	Bind `127.0.0.1`; reach via SSH or `tailscale serve`

TLS, by deployment mode (not all at once, the default needs none of the below):

Default tunnel path: Cloudflare terminates TLS at its edge; Traefik can stay plain HTTP internally.
Published-origin path: a Cloudflare Origin Certificate (or public-CA cert) + SSL/TLS mode Full (strict).
Private mesh HTTPS: tailscale serve or Tailscale certs.
ACME / Let's Encrypt: advanced, prefer DNS-01 when the origin shouldn't expose public port 80 (HTTP-01 validates on :80).

Two hard safety rules:

Never mount the raw Docker socket read-write. Front it with a read-only socket proxy and point Traefik's Docker provider at it, endpoint: tcp://dockerproxy:2375 on the shared proxy network, not /var/run/docker.sock. Expose only the endpoints the provider needs: label/event discovery requires CONTAINERS=1 and EVENTS=1, plus NETWORKS, INFO, and VERSION; add TASKS/SERVICES only if you run Swarm. Read access to the socket is still privilege, minimize it. The proxy itself is one small service:
```
dockerproxy:
  image: tecnativa/docker-socket-proxy
  environment: { CONTAINERS: 1, EVENTS: 1, NETWORKS: 1, INFO: 1, VERSION: 1 }
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro   # read-only
  networks: [proxy]
  restart: unless-stopped
```
Inject secrets at runtime. A clean pattern: override the container entrypoint with a tiny script that materializes cert/key files from env vars (umask 077, atomic mktemp+mv, chmod 600 the key), then execs the real binary. Keys live only in the running container, never in git or the image.

Maintenance hazard: file-provider routes can outlive their backend (a "dead route"). Periodically reconcile your routers against live backends, orphans are silent until someone hits the path.

6. Tailscale: reach admin tools without public listeners

The problem it solves. You need to reach SSH, dashboards, metrics, and databases, but exposing any of them publicly is exactly what you're trying to avoid.

How it fits. Tailscale is a WireGuard mesh VPN (a "tailnet"). Every machine joins the mesh and gets a stable private IP, from the CGNAT range 100.64.0.0/10, reachable only by other devices on the mesh, never from the public internet. SSH rides the mesh too: no public port 22, no bastion, no jump box.

*Bind-address cheat-sheet, where a service listens decides who can reach it.* This is the most concrete version of the two-plane rule:

The bind address is the boundary: loopback, mesh, LAN, or public internet.

Bind address	Who can reach it	Use for
`127.0.0.1`	Same machine only	Local-only admin; front with `tailscale serve` or SSH
`100.64.x.x` (tailnet IP)	Tailscale peers only	Mesh admin: dashboards, metrics, databases
`0.0.0.0`	Every interface, including public	Avoid for admin, the classic accidental exposure
`10.x` / `192.168.x`	LAN clients	Not the mesh, a different network entirely
Public IP	The whole internet	Only when deliberately publishing (prefer the tunnel)

Bring the box onto the mesh:

# Enroll the node (one login)
tailscale up

Two different ways to reach a service over the mesh, pick one per service, don't stack them:

tailscale serve (HTTPS front via your tailnet DNS name). This proxies a local port through tailscaled and presents a real mesh TLS cert on the node's MagicDNS name:
```
tailscale serve --bg 8080
```
Caveat: tailscale serve requires HTTPS certificates enabled in the tailnet (MagicDNS + HTTPS certs). On first use it opens an interactive web-UI consent flow to enable HTTPS, so it is not headless-friendly unless that's already been enabled by a tailnet admin. This is the only one of the two patterns that gives you a TLS cert.

Direct mesh-IP port binding (raw port, no TLS). Bind the container's port to the node's Tailscale IP so it's reachable on the mesh but nowhere else. This exposes the raw port directly, there is no mesh cert here.

The binding pattern, and its sharp edge. In Docker Compose you can bind an internal service to the mesh IP, never 0.0.0.0:

services:
  metrics:
    image: my-metrics:latest
    ports:
      # mesh IP, must already exist on an interface at container start
      - "${MESH_IP:-127.0.0.1}:9090:9090"

Important, this is not a graceful fallback. A Docker ports: binding to a specific host IP requires that IP to exist on an interface when the container starts. The :-127.0.0.1 default only fires when MESH_IP is unset or empty, not when it's set to a stale or currently-absent Tailscale IP. If MESH_IP is hardcoded in an env file and tailscaled is down or the address has changed, Docker fails the container with cannot assign requested address rather than quietly serving on loopback. Two robust options: (a) bind to loopback only and front the service with tailscale serve, which survives IP churn via the tailnet name; or (b) if you must bind the mesh IP, ensure it exists before container start and accept that a stale/missing IP is a hard failure. Prefer referencing the node by its MagicDNS name over pinning a raw IP in env, consistent with the stable-names rule below.

Make the rule a test, not a tribal convention. Parse ss -ltnp and fail CI if any policy-restricted port isn't bound inside {loopback, Tailscale mesh range}. Use a real CIDR membership check, a substring grep is too blunt and easy to invert:

# Fail if an admin port is NOT bound to loopback or the Tailscale mesh.
# Allow-list by CIDR membership, never a substring deny-list:
#   loopback        127.0.0.0/8
#   Tailscale mesh  100.64.0.0/10   (100.64.0.0-100.127.255.255)
# NOTE: 10.x / 192.168.x are private-LAN ranges, NOT the mesh, do not allow them.
in_cidr() {  # in_cidr <ip> <network> <prefix>
  python3 - "$@" <<'PY'
import ipaddress,sys
ip,net,pfx=sys.argv[1],sys.argv[2],int(sys.argv[3])
sys.exit(0 if ipaddress.ip_address(ip) in ipaddress.ip_network(f"{net}/{pfx}") else 1)
PY
}

bind_ip=$(ss -ltnp | awk '/:9090 /{print $4}' | sed 's/:[0-9]*$//' | head -n1)
if [ -z "$bind_ip" ]; then
  echo "OK: metrics port not listening"; exit 0
fi
if in_cidr "$bind_ip" 127.0.0.0 8 || in_cidr "$bind_ip" 100.64.0.0 10; then
  echo "OK: metrics is mesh/loopback only ($bind_ip)"
else
  echo "FAIL: metrics port publicly/LAN bound ($bind_ip)"; exit 1
fi

Two complementary controls, use both for sensitive services:

Tailnet ACLs decide which peers may reach which ports.
Bind-policy decides which interface a service listens on.

Stable names: use the mesh DNS (MagicDNS) names in config and scripts so they survive IP churn. For cross-host automation, prefer the fully-qualified name, bare short names don't always resolve.

*When not to use the mesh: a flow that needs a public callback (OAuth, webhooks) can't run on a raw-IP HTTP mesh endpoint, providers reject http:// and raw IPs. Those surfaces belong on the public plane with a real domain and TLS. The mesh is for humans-as-admins and server-to-server*.

The verification reflex: a 200 on localhost tells you nothing about public exposure. Check the actual bind address (ss -ltnp), not just that the service responds.

7. Putting it together: deploy a sample app end to end

Let's ship a real, copy-paste-runnable container, traefik/whoami, which just echoes the request it received, so you can see the chain working, at app.example.com.

1. Run it on the shared proxy network. The proxy reaches it by service name, not localhost (from inside the proxy, localhost is the proxy itself):

docker network create proxy      # once, if it doesn't exist
docker run -d --name whoami --network proxy --label traefik.enable=false traefik/whoami

2. Give it a Traefik router. Drop traefik.enable=false and add the router labels, here as a compose file you can keep. Note entrypoints=web (:80): the tunnel terminates browser TLS at Cloudflare's edge, so the app is plain HTTP inside the box.

# whoami.compose.yml
services:
  whoami:
    image: traefik/whoami
    networks: [proxy]
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`app.example.com`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
networks:
  proxy:
    external: true

docker rm -f whoami                          # remove the enable=false test container
docker compose -f whoami.compose.yml up -d   # bring it up with its router labels

3. Expose it publicly. If you're using a tunnel, the proxied CNAME was created by cloudflared tunnel route dns in §4 (don't hand-make a second one); add the matching ingress rule to config.yml. For the published-:443-origin path instead, create the proxied DNS record yourself and set SSL mode to Full (strict). No new public port, ever.

4. Add a second route to show fan-out. A /api PathPrefix router at priority: 10 to a backend, with the UI catch-all at priority: 1.

5. Verify the FULL chain, not just localhost:

# Real DNS, no --resolve pin: this exercises Cloudflare → tunnel → proxy → app
curl -I https://app.example.com   # expect 200
curl -I http://app.example.com    # 301/308 IF you enabled HTTP→HTTPS redirect
                                  # (Cloudflare or Traefik); otherwise 200

Recommended: turn on HTTP → HTTPS redirect for public hostnames, Cloudflare's Always Use HTTPS, or an explicit Traefik redirect on the published-origin path. Until you do, a 200 on plain http:// is expected, not a failure.

Note what the cert actually is: via the tunnel the browser sees Cloudflare's edge cert, so a green padlock here does not prove your origin cert is real, it could be Traefik's self-signed default and you'd never know. To validate the origin cert end to end you need the Full (strict) published-origin path, which actually checks it.

6. Now add an admin surface and keep it OFF the public plane. Bind the metrics dashboard to the mesh IP (or front it with tailscale serve), set traefik.enable=false (or give it a mesh-only ipAllowList router scoped to 100.64.0.0/10), and reach it only over Tailscale. The allowlist works here, and not on a public route, precisely because the request arrives directly over the mesh, so Traefik sees your tailnet IP (§5 caveat).

Deploy hygiene: recreating containers is what picks up changes. Flipping a release symlink alone does not restart them, docker compose up -d --force-recreate, and verify by Created time / image / mount, not just "healthy." Inject secrets at runtime so they never land on disk.

8. Authentication: which lock for which door?

"Self-hosted" doesn't mean "open." But the mistake beginners make isn't forgetting auth, it's reaching for one kind of auth and bolting it onto everything. A password prompt in front of an API that a script calls is useless friction; an OAuth login in front of a tool only you touch over the mesh is wasted effort. The skill is matching the lock to the door.

Two questions decide it every time:

Who's knocking, a human in a browser, a machine/script, or just you-the-operator?
What's behind the door, a public app, an internal dashboard, or a programmatic API?

You'll usually run several of these at once, that's correct, not over-engineering. The goal isn't to pick one lock; it's to put each one on the right door.

Match the lock to the caller: human, machine, operator, or app user.

1. The cheapest lock is the network itself. For anything only you administer, the mesh is the first gate: keep it off the public plane and scope a Traefik ipAllowList to 100.64.0.0/10 (§6). A service with no public listener can't be brute-forced, there's no door to pick. For a low-risk personal dashboard, Tailscale ACLs + bind-policy are often enough on their own. But network reach is access, not authorization: for anything destructive, sensitive, or multi-user, keep the app's own auth enabled too, and still change its default credentials. The goal isn't one lock; it's matching the lock to the blast radius.

2. Basic Auth, the one-line password wall. When a small, low-stakes tool is public and you just want a gate, Traefik's basicAuth middleware is the least-effort option. Generate a bcrypt credential and attach the middleware:

# htpasswd ships with apache2-utils; -B = bcrypt, -n = print (don't write a file)
htpasswd -nbB admin 'choose-a-strong-unique-password'
# → admin:$2y$05$REPLACE_WITH_YOUR_OWN_BCRYPT_HASH

# As a Docker label. NOTE: in docker-compose, every literal $ in the hash
# must be DOUBLED to $$ or Compose will try to interpolate it as a variable.
labels:
  - "traefik.http.routers.my-tool.middlewares=tool-auth"
  - "traefik.http.middlewares.tool-auth.basicauth.users=admin:$$2y$$05$$REPLACE_WITH_YOUR_OWN_BCRYPT_HASH"

# Or in the dynamic file (single $ here, no doubling outside compose):
http:
  middlewares:
    tool-auth:
      basicAuth:
        users:
          - "admin:$2y$05$REPLACE_WITH_YOUR_OWN_BCRYPT_HASH"

Know its limits before you trust it: Basic Auth sends a base64 credential on every request, so it is only safe when every hop carrying it is protected. On this guide's default tunnel path, the browser-facing hop is HTTPS (Cloudflare's edge) and the final cloudflared → Traefik hop is plain HTTP but stays private inside the box's Docker network, never the public wire; if you instead publish the origin directly, require HTTPS end to end (the websecure / Full-strict variant). It has no logout, no sessions, and usually one shared credential, so rotate it and don't put it alone in front of anything sensitive. It's a tidy gate for a staging tool or a webhook receiver, not an identity system.

3. Identity at the edge, SSO without writing login code. When real humans should authenticate with an identity (Google, email OTP, a company SSO), put an identity proxy in front of the route so an unauthenticated request never reaches your app. Two flavors:

Cloudflare Access (Zero Trust, free tier): define an application for app.example.com, attach an allow policy (e.g. "these emails, email-OTP, 24h session"), and Cloudflare challenges the request at its edge before the tunnel. Browser flows get a 302 to <team>.cloudflareaccess.com; API/service-token flows get 401/403. All of those mean gated, not broken.
Self-hosted forward-auth (Authelia, Authentik, or oauth2-proxy) when you'd rather not depend on Cloudflare for identity. Traefik calls your auth service on every request via a forwardAuth middleware:
```
http:
  middlewares:
    sso:
      forwardAuth:
        address: "http://auth:9091/api/verify?rd=https://auth.example.com/"
        trustForwardHeader: true
        authResponseHeaders: ["Remote-User", "Remote-Groups"]
```
Point any route at middlewares: ["sso"]; the auth service owns the login UI and returns 2xx (allow) or 401 (challenge). Your app stays login-free.

4. Tokens for machines. Browsers can do SSO; scripts, CI jobs, and webhook senders can't. Give them bearer or service tokens, deny by default, one token per caller, scoped and rotatable. Cloudflare Access issues service tokens for exactly this, a machine path behind the same edge gate, authenticated by a token header instead of a login. At the app layer, a tiny middleware that rejects any request without a valid Authorization: Bearer … is the same idea self-hosted. Whichever you use: never log the token, and rotate on a schedule.

5. App-layer login, for real accounts. If the app has its own users, roles, and sessions, that logic belongs in the app (sessions, JWT, or OIDC). Keep it even when an edge gate already fronts the route: the two are defense in depth, and an edge policy is one misconfiguration away from bypass. A fail-closed app-level floor means a slipped edge rule degrades to "logged warning," not "open door."

The rules that keep auth sane:

Stack gates on sensitive surfaces: edge identity → proxy ipAllowList → app auth. One wall is a single point of failure; layers survive a mistake in any one of them.
Deny by default. Never "secure by obscurity", an unguessable URL is not auth.
Credentials only over HTTPS. Basic Auth, bearer tokens, cookies, all of it.
Don't over- or under-engineer. No OAuth on a tool only you reach over the mesh; no bare Basic Auth in front of real user data.
*Bypass narrowly, and know what it costs. Bypass turns off Cloudflare Access enforcement and request logging for matching traffic. Health checks are usually fine to Bypass. Metrics should stay private (they leak internals), keep them on the mesh, don't Bypass them onto the public plane. Webhooks/automation that still need auth + an audit trail belong on a service token (Service Auth)* or an app-layer signature check, never a blanket Bypass.

9. Security checklist

Before you check a single box, re-ask the one question: Does the public need to reach this? If the answer is no, it should have no public listener at all.

The single most important lesson: never expose admin dashboards publicly, keep them on the Tailscale plane. Everything else is defense in depth.

[ ] Every public DNS record that fronts an app is PROXIED (orange-cloud). No grey-cloud records leaking your origin IP. (Tunnel cfargotunnel.com CNAMEs are proxied-only by design; the orange-cloud choice is yours to get wrong only on the published-origin path.)
[ ] If using origin certs with a published port: SSL mode is Full (strict), the only mode that validates the origin cert. Not Flexible, not plain Full.
[ ] Inbound firewall is closed except what you truly need. A tunnel needs zero inbound; a published origin needs only 443, ideally locked to Cloudflare's IP ranges.
[ ] Every admin/internal service binds to loopback or the Tailscale mesh IP (100.64.0.0/10), verified with ss -ltnp, not assumed. A CI bind-policy test enforces it with a real CIDR check, and 10.x/192.168.x LAN ranges are not treated as the mesh.
[ ] exposedByDefault=false on the proxy; only explicitly-labelled containers get a public router. Admin containers carry traefik.enable=false.
[ ] The Docker socket is read-only via a socket proxy, never mounted read-write into the proxy; the Traefik provider points at tcp://dockerproxy:2375, not the raw socket.
[ ] Secrets are never in git or the image, injected at runtime; cert/key files written 0600 by an entrypoint script at container start.
[ ] Change default credentials on every dashboard before it's reachable by anyone, including over the mesh.
[ ] *Public sensitive routes stack public-plane gates: Cloudflare Access / Service Auth → app-layer auth or token validation. Do not* rely on a Tailscale ipAllowList on a Cloudflare-fronted public hostname (Traefik sees the tunnel connector, not your tailnet IP).
[ ] *Mesh-only sensitive routes stack private-plane gates:* Tailscale ACLs → bind policy → optional Traefik ipAllowList → app auth, for destructive/sensitive/multi-user tools.
[ ] Treat tunnel ingress as config-as-code and reconcile it. A periodic diff of live config vs your declared routes catches an unexpectedly-exposed hostname before it's an incident.
[ ] Reconcile orphaned routes. A router pointing at a dead backend is a silent maintenance hazard.
[ ] Tear down a sensitive route in the right order: remove the public route (DNS + tunnel rule) and confirm the data plane no longer reaches origin before deleting the identity gate. An ungated live route is a breach; a stale gate on a dead route is harmless.

10. Advanced hardening once the base path works

You now have a defensible foundation. Once the happy path is solid, harden it in four directions:

Observe

Structured JSON access logs from the proxy with a field allowlist (drop headers by default), shipped to a log store you reach only over the mesh.

Deploy

A materialize → ship → flip release pattern with runtime secret injection, so deploys are atomic and rollback-able. Remember: flipping a symlink doesn't restart containers, docker compose up -d --force-recreate does.

Scale

A second box, route cross-host backends through the proxy's file provider: one front door, many machines.
More admin surfaces, layer Cloudflare Access policies as you add them, and keep the mesh as the floor for anything that doesn't need the public.

Audit

Reconcile orphaned routers (routes pointing at dead backends) and tunnel-ingress drift (a hostname exposed that you never declared), and enforce the bind-policy CI check so the two-plane rule is tested, not trusted.

Troubleshooting, the failure-mode tells worth knowing cold:

Symptom	Likely cause	Check
Redirect to login	Gated and healthy, not broken	`302` (browser) or `401`/`403` (API/service-token) all mean "gated"
Edge `404`, empty body, `server: cloudflare`	Missing tunnel ingress rule	the hostname never reached your origin
`502` at the edge	Tunnel reached origin, origin refused	Traefik entrypoint + the `service:` URL (the addressing fork, §4)
Works on `localhost` only	Not routed through the public chain	`curl` real DNS, not loopback
Admin app reachable publicly	Wrong bind or a stray `traefik.enable=true`	`ss -ltnp` + the container's labels
Green padlock but origin unverified	That's Cloudflare's edge cert	validate the origin cert via the Full (strict) path

Further reading: Traefik routers/middlewares and static-vs-dynamic config docs, Cloudflare Tunnel + Zero Trust docs (including cloudflared tunnel create/route dns), Tailscale ACLs, the 100.64.0.0/10 CGNAT addressing model, and tailscale serve, plus the Let's Encrypt/ACME challenge-type reference.

The discipline that makes all of this hold together is a single sentence: classify every service by whether the public needs it, put it on the matching plane, and verify the bind address, not just the health check. Do that consistently and your self-hosted estate stays sane no matter how many apps you add.

Verify the Verifier

Wed, 24 Jun 2026 07:00:00 +0000

A verification agent is still a single source. The two-tier pattern, generate cheap, then run an adversarial skeptic that checks each claim against live state, earns its keep when the skeptic catches the draft's errors. It inverts the moment the skeptic introduces its own. The asymmetry that makes the pattern work, assume the draft is wrong, is the same one that makes a wrong correction dangerous: the architecture has pre-committed to trusting the second voice, and nothing downstream is positioned to doubt it. The second pass was the doubt.

The tell is cheap to check and easy to skip. A confident "CONFIRMED, corrected" that flips a premise the draft had right is not a resolution; it is a flag. Read the skeptic's cited evidence: does it name the artifact under test, or a sibling that resembles it? Mismatched-but-confident is the signature of a backside error one layer up.

A correction earns its way into the output the way authority does, on proof, not standing trust. Promote it on a citation that names the right thing, on agreement across the fan-out, or on one direct probe you ran yourself. A verdict from an unverified verifier is theater.

The Gate Is the Product

Tue, 16 Jun 2026 07:00:00 +0000

When a fleet of agents produces eight sprints of work in two days, the instinct is to celebrate generation. But the abandoned drafts pile up by the hundreds, and the real scarcity reveals itself: a human authorizes and converges only so much. Generation was never the constraint. The merge-authority gate, and the labor of reconciling what was generated into something coherent, is.

That reframes what the product is. The gate is the product, the thing that decides what crosses from generated into shipped, and it earns trust by shipping shadow-first, observing what it would have done before it acts. Authority follows the same rule: delivered on demand at the moment of decision, never a bearer credential left sitting at rest. The model is replaceable. The gate is the part that lasts.

Open-Source the Gate, Not the Verdict

Tue, 09 Jun 2026 07:00:00 +0000

It is tempting to think the intelligence is the product, that a sharper reviewer model is what makes AI code review safe. It is the opposite. The model that reads a diff is replaced within months. The thing that governs is the gate around it: the deterministic logic that default-denies on an unparseable or contradictory verdict and leaves a receipt of what it decided.

So the gate is what you open-source and the gate is what does the governing. The model sits behind a command seam and is swapped without touching the enforcement around it. Publishing the verdict ties governance to a model version; publishing the gate gives you something that holds as the models churn underneath it. The durable artifact is the boundary, not the brain.

The Authority Gate, Made Runnable: Fail-Closed Merge Admission for Agent-Written Code

Mon, 08 Jun 2026 07:00:00 +0000

Part I, The Gate, Made Runnable

Every argument for runtime governance eventually meets the same objection: show me. The pre-execution authority gate is a clean idea, evaluate authority before state mutation, halt on ambiguity, never fall back to probabilistic scoring, but an idea is not an artifact. failclosed is the artifact: the authority gate applied to one narrow, high-frequency mutation boundary, the merge.

The choice of boundary is deliberate. A merge is small, concrete, and constant, and it is where AI-written code crosses from proposal into production state. It is also where the current generation of tooling fails in a specific, instructive way.

Part II, Why Advisory Review Fails Open

The dominant pattern in AI code review relays the model's own verdict. A reviewer model reads a diff, decides whether it looks acceptable, and the surrounding tool treats that decision as the gate. When the model says "looks good," the change is mergeable. This is advisory governance, and advisory governance fails open by construction.

Consider the failure that matters most: the reviewer's output is not clean. It is truncated by a token limit, malformed by a formatting slip, or internally contradictory, a verdict of "needs fixes" attached to a findings list where not one finding cites a file or a line. A human reading that output distrusts it immediately. An advisory tool does the opposite. With no clean verdict to relay, it defaults to permitting the merge, because permitting is the path of least resistance and no deterministic rule forces a stop. The reviewer becomes a single point of silent failure precisely when its output is least trustworthy.

The cost is not abstract. A merge is a state mutation. An ungoverned merge of agent-written code is an irreversible state change with no attestation of who, or what, authorized it. In a regulated environment, that is audit indefensibility, the inability to prove, after the fact, that a change was admitted under an enforced rule rather than a model's good mood.

Part III, Distrust the Reviewer

failclosed treats the reviewer the way the control plane treats an agent: as an untrusted compute node whose output is inadmissible until proven. It runs the reviewer, and then it refuses to trust the verdict.

The output passes through a deterministic parser. The gate refuses to report MERGE_READY when that output is unparseable, schema-invalid, or self-contradictory. Take the contradictory case: verdict "fix," no finding citing a file. An advisory tool, finding no actionable findings, reports the change clean and admits it. failclosed reads the same output, recognizes that a "fix" verdict with no located finding is an indeterminate state, and halts. Ambiguity translates to a stop. There is no fallback to "probably fine," because a fallback to "probably fine" is the precise mechanism by which ungoverned actions execute.

This is the posture of the pre-execution authority gate, narrowed to the merge boundary: enforcement precedes the mutation, the default is deny, and a merge that cannot be deterministically evaluated does not proceed. The reviewer's confidence is an input to be checked, not an authority to be obeyed.

Part IV, Receipts Over Logs

A decision that cannot be inspected is not enforcement; it is a side effect. Every failclosed run emits a structured, inspectable record, what was reviewed, what failed the gate, and why the final status is what it is. This is attestation, not telemetry. A log says an event happened. A receipt says a decision was made, on this input, under this rule, with this outcome, and it can be read back and reproduced.

That distinction is what makes the gate auditable. An auditor does not have to trust that failclosed ran, or that it ran correctly. They read what it decided. The verdict and its grounds are durable artifacts, not transient console output discarded after the build.

Part V, The Boundary Is the Example, Not the Point

The merge is a small case, chosen because it is public and runnable in two minutes. The principle does not depend on the case. Anywhere an AI system's output gates a state mutation, a deployment, a payment, a configuration change, a database write, the same structure holds. The model proposes. A deterministic gate disposes. The gate distrusts the model's own confidence and defaults to deny when the input is ambiguous. Capability is removed at the boundary, not requested politely from a system free to ignore the request.

failclosed is not a novel reviewer; better reviewers keep arriving. It is the discipline around the reviewer, the parser, the fail-closed gate, the receipt, and that discipline is the part that does not depend on which model reads the diff this quarter. The model is replaceable. The gate is the product.

That is the difference between governance and hope. failclosed is public and runnable: github.com/OrionArchitekton/failclosed. The same enforcement model, applied across an enterprise agent stack rather than a single merge gate, is a Runtime Governance Readiness Scan at danmercede.com.

A Governance Claim You Cannot Run Is Advisory

Tue, 02 Jun 2026 07:00:00 +0000

Every strong governance argument eventually meets the same question: show me. A position paper describes what enforcement should do. It does not establish that any system does it. Until the claim reduces to something a reader runs, and watches halt on the case it promised to halt on, it remains advisory, no matter how tight the prose.

The proof obligation for runtime governance is an artifact, not an essay. A skeptic clones it, feeds it the ambiguous input, and sees it default-deny. Doctrine earns its authority by becoming executable: the distance between a claim about enforcement and an enforcement you run is the distance between content and proof.

Distrust the Model's Own Output Shape

Tue, 26 May 2026 07:00:00 +0000

A model that decides whether the next step runs is part of the control path, and its output is input from an untrusted source. It arrives wrapped in markdown fences, truncated by a token limit, carrying a verdict that contradicts its own findings. A gate that reads that output as if it were a clean signal has quietly relocated the enforcement into the model's good behavior.

The boundary has to do the unglamorous work: strip the formatting, validate against a schema, and on anything malformed or self-contradictory, refuse rather than guess. The model proposes; the deterministic gate disposes, and it disposes by parsing, not by trusting. Distrusting the shape of model output is not pessimism, it is the only place the control actually lives.

Pre-Execution Authority Gates: Why Governance Must Be Evaluated Before State Mutation

Wed, 20 May 2026 07:00:00 +0000

Part I, Runtime Governance Engineering

The runtime governance control plane begins with a non-negotiable premise: governance cannot operate as a post-hoc advisory function. Advisory governance depends on observability layers and probabilistic alerts triggered after state mutation has already occurred. This architecture accepts that unauthorized actions can execute before intervention. That latency is systemic risk. Governance evaluated after execution is telemetry. Telemetry is not enforcement. To mitigate systemic risk, governance must exist as a deterministic constraint evaluated strictly before any state mutation occurs.

When an execution framework treats governance as an external observer rather than a foundational constraint, it inevitably permits untrusted compute nodes to initiate actions that must subsequently be mitigated. True governance demands deterministic enforcement at the execution boundary, ensuring that an agent physically cannot execute a disallowed command. By shifting the evaluation entirely to the pre-execution phase, the system guarantees that no network packet leaves the agent enclave and no database transaction is initialized without explicit cryptographic authorization. The intelligence layer determines its intended action, but the physics of the control plane dictate what is physically possible to execute.

The architecture enforces a non-bypassable control plane. This means the execution substrate physically and cryptographically lacks the network routing, identity credentials, and memory pathways to interact with external APIs or databases directly. To enforce this, AI models and agents are treated as untrusted compute nodes. They are strictly isolated in ephemeral, lane-isolated memory enclaves, implemented natively as distinct namespaces or microVMs.

Within these hardware-isolated lanes, agents are physically incapable of resolving external DNS or establishing direct outbound TCP connections. All tools provided to the agent are actually internal RPC stubs. When an agent attempts an action, it merely passes a structured intent payload to the RPC stub, which forwards it to the control plane. Because the execution node is stripped of routing primitives and external credentials, the authority gate becomes structurally non-bypassable. Every intended action must traverse the control plane, where it is subjected to deterministic evaluation. Enforcement occurs precisely at this juncture: at the structural boundary between the execution substrate and external state or API layers.

The Runtime Governance Control Plane is the sole routing and enforcement boundary. It is explicitly designed to be devoid of probabilistic reasoning; it operates purely on deterministic logic. The Pre-Execution Authority Gate inspects the intended state mutation payload. To ensure that human intent is enforced with sub-millisecond evaluation latency at runtime, high-level human policies are compiled into low-level deterministic rules. These execution roles map to Attribute-Based Access Control (ABAC) and Role-Based Access Control (RBAC) constraints defined in the signed policy.

The validation engine architecture isolates the request payload and evaluates it against the loaded, compiled policy constraints in memory. Simultaneously, signature verification mechanisms verify the identity signature of the requesting agent and check it against the active session ledger. Evaluation logic within this engine is strictly binary. The architecture maintains a default-deny posture; if the request lacks necessary context, or if the policy does not explicitly permit the action, the state is flagged as indeterminate. Ambiguity translates directly to an execution halt. There is no fallback to probabilistic scoring or secondary inference models. The compiled policy must explicitly match the intended intent, or the action is denied at the threshold.

When validation fails, enforcement is instantaneous and absolute. Enforcement occurs synchronously in memory at RPC ingress. On failure, execution halts instantly. The payload is discarded before routing is ever established. If ambiguity exists in either the execution intent or the policy mapping, the system defaults to a hard execution halt. The state mutation request is dropped, and the execution thread is suspended or terminated.

Any attempt to bypass the RPC stubs and communicate directly with external IP addresses results in packet drops at the container boundary. Furthermore, if a halt is triggered, the engine freezes the agent's memory state and opens an escalation pathway, routing the frozen context back to the human authority layer for review. This ensures that the agent cannot autonomously retry or attempt to subvert the blocked request; the execution thread remains cryptographically locked in its frozen state until explicitly unblocked or destroyed by the human-defined escalation protocol.

A deterministic enforcement system requires immutable cryptographic proof that a policy was evaluated and either permitted or blocked. At the Authority Gate layer, the artifact that proves enforcement is a trace log detailing the specific boolean evaluation failure. If an execution boundary bypass is attempted, substrate network egress rejection logs serve as the proof of enforcement.

For deeper state tracking, the immutable receipts service guarantees that every material action, whether approved or denied, generates a receipt. This architecture provides absolute non-repudiation. Receipts are written to an append-only ledger using cryptographic hash chaining, such as Merkle trees. The receipt journals the state mutation by including the agent's exact request payload, the specific policy version evaluated, the cryptographic signatures of both, and the timestamp. For a denied request, the artifact that proves enforcement is the cryptographic hash generated by the Immutable Receipt Service. Because the system requires synchronous receipt generation before finalizing the mutation, no action can occur off the books. If the receipt cannot be written to the ledger, the transaction is rolled back and execution halts. Enforcement is not inferred. It is cryptographically attested by the chained ledger entry representing the constrained state transition.

If governance is not deterministically enforced before state mutation, it is not governance. It is hope.

The Merge Is a State Mutation

Tue, 19 May 2026 07:00:00 +0000

Teams guard runtime mutations carefully and then merge to main on a glance at a checkmark. But a merge is a state mutation, often the most consequential one, the moment a change crosses from proposal into what runs. If the gate in front of it cannot conclusively resolve and still lets the merge through, the release boundary is governed by optimism.

The merge gate is an authority gate at a different altitude. Its check evaluates the actual merge state, and on anything it cannot resolve, a degraded signal, an ambiguous result, a missing required check, it halts. Default-deny does not stop at runtime; it belongs at every boundary where intent becomes durable state, and the merge is one of them.

Posted Is Not Published

Tue, 12 May 2026 07:00:00 +0000

A POST that returns 200 means the platform accepted bytes. It does not mean a human will see the content, that it cleared review, or that it was not silently dropped. A pipeline that closes its receipt at "posted" is recording its own intent and calling it an outcome. Posted is a claim; published is a verified fact, and the two are not the same event.

The receipt's terminal state has to be earned: an independent check that the content is actually visible, or an explicit, documented abort when the channel cannot be verified. An unverifiable post does not get an assumed-success record, it terminates as aborted against a closure note. Enforcement here is refusing to call a thing done because the easy signal said so.