Notary

The problem

Data catalogs accumulate claims about units, freshness, completeness, enums, and deprecation, and nothing checks whether they are still true. A column labeled USD that actually stores integer cents makes an agent quote revenue 100x off, with the catalog’s authority behind it. As agents inherit catalogs as ground truth, a stale claim becomes an amplifier: the downstream answer carries the lie with the catalog’s confidence.

Point Notary at a DataHub asset and it runs a verification loop. It reads the asset’s live descriptions from DataHub, extracts typed, checkable claims behind strict entailment gates (an unstated claim never reaches a probe; a failed extraction is disclosed, never guessed), then probes the warehouse with bounded, read-only, capped SQL. It adjudicates each claim CONFIRMED, CONTRADICTED, or UNVERIFIABLE with evidence, and writes back through the DataHub MCP Server a trust ledger, an evidence dossier, a provenance-labeled corrected description, and an incident on dangerously-wrong high-usage assets. Every write is reversible.

Quickstart

No API key needed: extraction replays captured completions by default. Python with uv.

Clone and install:

git clone https://github.com/OrionArchitekton/notary
cd notary
uv venv && uv pip install -e '.[dev]'

Command surface

• notary.eval: Regenerates the honest evaluation table with no DataHub required: 9 of 12 planted lies caught, 0 of 6 adjudicated controls misclassified, published verbatim with misses included.
• notary.run: The full verification loop against a DataHub asset: read the live descriptions, extract checkable claims, probe the warehouse read-only, adjudicate with evidence, and write back the trust ledger.
• notary.rollback: Reverses every write Notary made to DataHub: trust ledger, evidence dossier, corrected description, and any incident, so a verification run is never a one-way door.
• hosted replay: A frozen, reproducible bundle at notary-replay.vercel.app: the whole verification story renders with nothing generated on load and no credentials.
• test suite: All 7 spec scenarios are demonstrated by the tests; 10 are live DataHub integration round-trips (skipped without a local quickstart), the rest deterministic replay and eval.

Why it is different

• The model proposes, the gate disposes: Claims are extracted behind strict entailment gates: an unstated claim never becomes a probe, and a failed extraction is disclosed rather than guessed. The language model only proposes what is checkable; deterministic gates and read-only probes decide what is true.
• Bounded, read-only probes with evidence: Every verdict is grounded in a bounded, read-only, capped SQL probe against the warehouse. Each claim is adjudicated CONFIRMED, CONTRADICTED, or UNVERIFIABLE and carries an evidence dossier: the probe SQL, the measured value, and the claim diff.
• Honest evaluation, misses included: The evaluation is published verbatim: 9 of 12 planted lies caught and 0 of 6 adjudicated controls misclassified. A wrong CONTRADICTED is treated as worse than a declared miss, so contradictions are issued only under corroboration and the failures are printed alongside the wins.
• Every write reverses: Notary writes back a trust ledger, an evidence dossier, a provenance-labeled corrected description, and an incident on dangerously-wrong high-usage assets, all through the DataHub MCP Server. Because every write is reversible with notary.rollback, running it against a live catalog is never a one-way door.

Links

• GitHub repository
• Demo video
• Try it live
• Dan Mercede