Skip to main content

Proctor

Behavioral regression testing for AI agents that never give the same answer twice.

Open source (MIT) · TypeScript · 77 unit + 1 durable integration test, all green · UiPath AgentHack 2026 finalist.

The problem

An AI agent rarely produces byte-identical output twice, so assert-equals regression testing either flakes constantly or passes on garbage. The moment someone swaps the model or edits a prompt, behavior can drift silently, and no traditional test catches it. Teams are shipping agents into production with no regression safety net.

Proctor learns each agent's behavioral contract from sample runs: structural, exact, and semantic field assertions plus hard domain invariants like line items summing to the total. On every model or prompt change it re-checks the contract and classifies the drift three ways: real regression, legitimate evolution, or flaky. Consequential changes suspend a durable workflow on a human approval hook that survives restarts and multi-day review gaps; approved evolution versions the contract. It never rewrites its own tests silently.

Quickstart

Everything runs keyless by default: no API key, no UiPath credentials. Node 20+ and pnpm.

Clone and install:

git clone https://github.com/OrionArchitekton/proctor
cd proctor
pnpm install

Command surface

  • pnpm demo: Keyless narrated run: learns the invoice contract, passes the good model, catches the degraded one, opens the approval hook.
  • pnpm dev: Next.js dashboard: contract panel, run-a-change controls, live governance feed, approve/reject on a pending finding.
  • pnpm test: 77 unit tests across the engine (learner, runner, classifier, risk, store), shared schemas, the invoice SUT, and the UiPath gateway.
  • pnpm test:int: Durable pause/resume integration test: the workflow genuinely suspends on the approval hook and resumes, verified against the runtime event log.
  • live-gateway-probe: With UiPath credentials set, exercises the real REST gateway: starts an Orchestrator job, creates an Action Center task, publishes a test report to a queue.

Why it is different

  • Contracts, not golden outputs: The system under test is an LLM; it gives a different answer every run. Proctor asserts properties: tiered field assertions plus hard invariants that hold regardless of baseline, so the checks that guard money and identity never depend on fuzzy matching.
  • A three-way verdict: Most testing splits results into real versus flaky. Proctor adds legitimate-evolution: the output changed but is still correct, so it proposes a contract patch instead of filing a bug. A model upgrade that is genuinely better stops looking identical to a regression.
  • A durable human gate: Consequential verdicts suspend a durable workflow (Vercel Workflow DevKit) on an approval hook that survives restarts and multi-day reviewer gaps. Approval versions the contract; rejection changes nothing. Every decision leaves a receipt in the governance trail.
  • Verified live, reproducible keyless: The UiPath gateway ran against a real tenant with the shipped code: an Orchestrator job, Action Center approval tasks, and queue-published test reports. And the whole cycle reproduces on your machine with no keys at all: pnpm install, pnpm demo.

Links