# RECEIPTS — every publicly quoted number → its exact artifact

Date: 2026-08-08. Purpose: the 360 reviews asked for "the exact artifacts behind the
published numbers." The retired methodology page used to bind numbers to files; this
document is the durable in-repo binding; the PUBLIC mirror of the principal artifacts is https://portmem.com/evidence.html. Every path is in this repository (the retired
methodology page is vendored at `docs/archive/methodology-2026-07-21.html`).

## 1. Version governance (deck slide: "Same model + PortMem")

Model both columns: claude-haiku-4-5, same judge per harness. "Alone" = knowledge-only
(NO documents), re-run 2026-07-21 (`paper4/run_knowledge_only_baselines.py`, commit
4bcb17c); "with PortMem" = Track-A runs 2026-05 (`paper4/test_*_vs_llm.py`).

| Deck row | Alone | With PortMem | Artifacts |
|---|---|---|---|
| FDA recall lookups, n=100 | 29% | 99% | `paper4/data/fda_knowledge_only_baseline.json` · `paper4/data/portmem_vs_llm_claude_haiku_4_5_track_a_only.json` · logs `paper4/data/fda_apple2apple_*_run.log` |
| SCOTUS overruling pairs, n=100 | 48% | 99% | `paper4/data/scotus_knowledge_only_baseline.json` · `paper4/data/scotus_vs_llm_legal_kb_track_a_only.json` · log `paper4/data/scotus_apple2apple_hybrid_run.log` |
| FASB ASC supersessions, n=20 | 95% | 100% | `paper4/data/fasb_knowledge_only_baseline.json` · `paper4/data/fasb_vs_llm_legal_kb_track_a_only.json` · log `paper4/data/fasb_legal_kb_run.log` |

Caveats that must travel with the slide:
- The two columns are different runs months apart (May vs July 2026); same model and
  judge, different code states.
- A SECOND baseline exists and is less flattering: full-corpus-paste long-context
  (`paper4/run_trackb_baselines.py` → `paper4/data/{fda,scotus,fasb}_llm_alone_baseline.json`):
  FDA 80%, SCOTUS 96%, FASB 100%. The deck's "no documents" label is accurate but the
  full-corpus condition narrows the SCOTUS margin to 99-vs-96 and erases FASB's. The
  archived methodology page disclosed both columns
  (vendored in-repo at `docs/archive/methodology-2026-07-21.html`); anything public that
  quotes the trio should keep that disclosure within reach.

## 2. "~210 human-checked claims, zero false alarms, zero planted errors approved"

Multi-domain dogfood protocol + results: `eval/DOGFOOD_DOMAINS.md`; per-domain raw
reviews with `seeded_scoring` in `eval/dogfood/<domain>/review_{clean,corrupted}.json`
(FDA_ELIQUIS_Label, Apple_424B2_Notes_2025, EU_AI_Act_Ch_I-III, Table_Claims), ground
truth in each `errors.json`, corpora in `eval/dogfood/corpora/`. Ambipar (finance,
Portuguese) line: memory-of-record runs summarized in `eval/DOGFOOD_DOMAINS.md`'s
relation note. Known asterisks (documented in the same file): the EU AI Act pre-guard run had 1
wrongly-approved seeded error — the site's "one miss" line refers to it — and ALL
seeded catch rates were re-scored 2026-08-08 with a corrected diff-token matcher after
an audit found the original scorer inflating the EU cells (EU sized: 2/6 caught 4
unmatched, not 4/6; combined entered-review record: 11 of 12 flagged, 1 wrongly
approved). The zero-false-contradiction and zero-value-error-approved claims were
unaffected — they never depended on the matcher.

## 3. Table catch rate (new, 2026-08-08)

`eval/DOGFOOD_DOMAINS.md` §Table_Claims: sized extraction 4/4 seeded table errors
caught (wrong cell / wrong column / wrong row / wrong derived total), 0 false-supported,
0 clean false alarms. Raw: `eval/dogfood/Table_Claims/review_{clean,corrupted}.json`.

## 4. Speed vs an LLM agent (new, 2026-08-08 — read before quoting ANY speed claim)

`eval/dogfood/SPEED_BENCHMARK.md` + `eval/dogfood/speed_benchmark_results*.json`:
at single-document scale the site's "fraction of the time" wording is NOT supported
(a long-context pass is faster than PortMem); PortMem's demonstrated edge is
0 false-supported in every run. Corpus-scale paired run not yet done.

## 5. Retrieval SOTA / papers (research lineage, not product claims)

MuSiQue 0.8316 / 2Wiki 0.9527 / HotpotQA 0.9875 and the P2/P3 published results:
`EVIDENCE.md` and `ARTIFACTS.md` (benchmark→artifact map), `product1/PARITY.md` (the
live-vs-cached bridge parity: live 0.9525 == cached), arXiv 2603.28886 / 2604.03384 /
2604.09019 / 2604.14488 with runs in the respective paper repos/folders.
(`eval/phase2_nebius_*_results.json` holds the LIVE DENSE BASELINES, not the SOTA
numbers — a prior version of this file mis-bound them.)

## 6. Operational proofs (2026-08-08 drills)

`ops/drills/`: E2E customer walkthrough (8/8 steps on prod), load test (200/200 @
conc 24, p95 5.8s, limiter sheds with 429s), restore drill (RTO 8m53s, PITR), evidence
bucket no-delete probe (all 6 outcomes), invite-flow transcript (all checks).