Evidence

Every number PortMem publishes, bound to its raw artifact: dataset, sample size, model, code commit, raw output, scorer, caveats, and correction history. Files are served from this page unmodified; the repository commit for the runs below is 95d56fb37eeb.

Corrections are part of the record: where a number changed, the superseded value and the reason are stated, not deleted.

1 · Version governance (the 29→99 / 48→99 / 95→100 table)

Model both columns: claude-haiku-4-5. "Alone" = the model answering from its own knowledge, no documents (July 2026 re-run). "With PortMem" = Track-A runs, May 2026, same model and judge, different code state (disclosed). Harness: test_portmem_vs_llm.py family; baselines: run_knowledge_only_baselines.py.

Caveat that travels with this table: a second, less flattering baseline exists — the same model with the entire corpus pasted as context scores 80% / 96% / 100% (fda · scotus · fasb). PortMem's margin is largest where the corpus cannot be hand-fed to the model. The two columns are runs months apart; n=20 on FASB is small.

2 · Verification record (campaign 2026-08-09: 255 seeded checks, 1,478 clean claims)

Pre-registered whole-pipeline campaign (protocol frozen in git before the runs): 4 domains × 3 memos, 6 seeded errors × 3 repeats + seeded omissions. Extraction recall 0.79-1.00 (95% CIs), catch-of-extracted 0.85-1.00, clean false alarms 2/1,478. This campaign retired our earlier "zero planted errors ever approved" claim: measured false-supported rate 15/255 (~5.9%). The first publication of this section quoted 6/255; re-running the Apple cells with correct source scoping (the original Apple runs had silently verified against the whole corpus, a harness defect we disclose here) raised it, and per our corrections policy the superseded number stays visible. Completeness (B) was redesigned after the campaign flagged it; seeded-deletion recall is 19/36 (0.53) on the development set. A fully blind held-out campaign (4 never-seen corpora, protocol frozen before the run) followed: false-supported 8/243 (3.3%), clean false alarms 1/1,327, extraction 0.70-1.00 — the verification stages generalize — but completeness recall fell to 8/36 (0.22) blind, so roughly half of B's measured gain was fitting to the development memos. We publish that number rather than the flattering one: completeness is a disclosure surface today, not a recall guarantee. Full comparison: HELDOUT_RESULTS.md; raw runs under /evidence/heldout/campaign/. Full tables + per-seed audit: DOGFOOD_DOMAINS.md; raw runs: summary.json + per-memo JSONs under /evidence/dogfood/campaign_2026-08/.

Earlier record (~210 claims, superseded by the campaign above)

Four domains, memos written and error-seeded by the model against public or synthetic corpora, declared machine-verifiable ground truth (not human-labelled — stated plainly). Protocol + all runs: DOGFOOD_DOMAINS.md. Scorer: dogfood_domain.py (score_seeded).

• Clean-memo false contradictions: 0 across ~210 reviewed claims (every domain, both runs).
• Planted value errors (dates, rates, amounts, entities) wrongly approved: 0, ever.
• Table-bound errors (wrong cell / column / row / derived total): 4/4 caught at sized extraction, 0 false alarms (raw).
• Correction history: seeded catch rates were re-scored on 2026-08-08 after an audit found the original matcher over-crediting: end-to-end catch is 11/21 (previously published 13/21); of the 12 errors that entered review, 11 were flagged and 1 count-aggregation error was approved before its mitigation shipped. 9 seeds were never extracted as claims — the current extraction-coverage limitation, published in full in the speed benchmark below.

Raw per-domain artifacts: FDA · Apple 424B2 · EU AI Act · EU pre-guard; corpora in /evidence/dogfood/corpora/; each directory carries the seeded ground truth (errors.json) and the clean-memo run.

3 · Speed benchmark (a negative result, published)

We benchmarked PortMem against LLM-agent verification on a single 145KB document and the result went against us: a claim-by-claim agent loop matched our wall-clock and beat our catch rate, because our claim extractor under-sampled list-heavy text. The full write-up, including why our former public speed wording was removed: SPEED_BENCHMARK.md · harness · raw: 25-claim, 40-claim.

What PortMem held in every run: zero planted errors wrongly approved, every verdict carrying cited evidence, calibrated abstention, and the signed record — none of which the agent transcript has. Corpus-scale comparison (documents exceeding any context window) not yet run; no speed multiplier is quoted anywhere until it is.

4 · Full receipts map

The complete number→artifact index, including operational drill artifacts (restore, load, cross-tenant isolation, end-to-end walkthrough): RECEIPTS.md.