# Paired speed + catch-rate benchmark: PortMem vs LLM-agent verification

**Date:** 2026-08-08 (re-run same day with the corrected diff-token scorer after the
eval audit found the original matcher inflating catch rates — see
`eval/DOGFOOD_DOMAINS.md` scoring-rule note; these tables REPLACE the first
publication of this file, whose numbers used the defective matcher and whose run-1
transcripts were overwritten). · **Harness:** `eval/speed_benchmark.py`
(`--max-claims`, `--out`; per-seed detail + PortMem's full claim list persisted in the
result JSONs; transcripts suffixed per run). · **Judge everywhere:** prod-parity
routing chain (Haiku 4.5). · **Task:** verify the EU AI Act corrupted memo (6 seeded
errors, declared ground truth) against the 145KB EU AI Act Ch. I-III corpus,
faithfulness only. The agent sides get the full document handed to them in-context
free; a real agent must first find its context, and none of that cost is billed here.

| Side | 25-claim run | 40-claim (sized) run |
|---|---|---|
| PortMem review | 96.2s · caught 1/6 · 0 false-supported · 5 unmatched | 107.0s · caught 2/6 · 0 false-supported · 4 unmatched |
| Agent, 1 long-context pass | 14.8s · caught 5/6 · 0 false-supported · 1 unmatched | 23.6s · caught 2/6 · **1 false-supported** · 3 unmatched |
| Agent, claim-by-claim loop | 58.8s · caught **6/6** · 0 false-supported · 0 unmatched | 94.5s · caught **6/6** · 0 false-supported · 0 unmatched |

(caught = seeded error verdicted contradicted/uncertain; false-supported = a planted
error approved — the unsafe failure mode; unmatched = the corrupted statement never
appeared in a judged claim, i.e. it was never reviewed.)

## Update (same day): the extraction-coverage fix landed and recovered recall

Finding 2's diagnosis drove a product change: `review()` now defaults to CENSUS
extraction (section-by-section scan, claim count tracks document depth, coverage map
in the payload). Re-run on the same corrupted memo (`speed_benchmark_results_census.json`):

| Side | wall | caught | false-supported | unmatched |
|---|---|---|---|---|
| **PortMem (census)** | 126.4s | **5/6** (incl. the count seed, flagged uncertain by the count guard) | 0 | 1 |
| Agent 1-pass (same run) | 20.1s | 4/6 | 0 | 2 |
| Agent loop (same run) | 86.9s | 6/6 | 0 | 0 |

Catch-rate parity with the careful loop is nearly recovered (5/6 vs 6/6) at ~1.5x its
wall-clock, judging 200 claims vs 40. The speed conclusion is unchanged: no speed
claim until the corpus-scale run.

## Honest findings — read all of them before quoting anything

1. **The speed claim is refuted at single-document scale.** The sequential agent loop
   matches PortMem's wall-clock; the single long-context pass is 4-7x faster. Nobody
   should quote "a fraction of the time an LLM agent takes" for a document that fits
   in a model's context window.
2. **On this benchmark the naive sequential agent BEAT PortMem on catch rate — both
   runs, 6/6 vs 1-2/6 — at equal-or-better speed.** Stating it plainly because the
   first version of this file buried it. The mechanism is diagnosable from the
   persisted claim lists: PortMem's claim extractor spent its budget enumerating the
   memo's prohibited-practices LIST as individual items and never extracted 4 of the
   6 corrupted statements (including the "Nine categories" count seed, which
   dissolves when the list is itemized), so they were never judged. The agent's flat
   claim listing extracted all 6. **Product lesson, actionable:** extraction coverage
   is the weak link on list-heavy documents; a coverage-guided extractor (spread
   claims across sections; keep aggregate/count claims alongside their itemizations)
   is the fix, and B-side completeness already has the section map to drive it.
3. **What PortMem uniquely held, here and everywhere else: zero false-supported.**
   Across every PortMem run in this file and the whole dogfood suite, no planted
   error was ever approved; the fast agent pass approved one (sized run). But on this
   benchmark that safety margin over the agent LOOP is zero (it also never
   false-supported), so safety alone does not differentiate against a careful loop at
   this scale.
4. **What this benchmark does not measure** — the things a buyer actually pays for:
   per-claim cited evidence, calibrated abstention, version/supersession governance,
   the signed gate-and-ledger record, completeness (B-side), and behavior on corpora
   that exceed any context window. The agent transcript carries none of that. Those
   are real but UNMEASURED here; claiming them as offsets to this table requires
   their own benchmark.
5. **Corpus scale untested.** The Ambipar corpus (~1.5MB, 3 instruments) does not fit
   any context window: agent_1pass is impossible there and agent_loop must add its
   own retrieval. That paired run has not been done; until it is, no speed multiplier
   may be quoted anywhere.

## Implication for the site's Speed tile (founder decision, flagged)

The tile's wording is refuted at single-document scale and unproven at corpus scale.
Options: rescope to what the data backs (calibrated verdicts, nothing wrongly
approved, signed record — with the extraction-coverage fix landed, catch-rate parity
is recoverable), or run the corpus-scale benchmark first.

Artifacts: `speed_benchmark_results.json`, `speed_benchmark_results_sized.json`
(each with `seeded_detail` + PortMem's full claim list), `speed_agent_*_raw*.txt`.

## Corpus-scale run (2026-08-09, protocol §corpus-scale, amendment documented in eval/corpus_scale.py)

The missing half of the story. Ambipar indenture corpus, **1.78MB** (no context
window holds it), fresh memo + 8 seeded errors, same judge chain on both sides;
the agent scans the corpus in 140K-char windows per claim and pays its real
search cost (agent claims capped at 15, disclosed).

| Side | wall | provider calls | caught | false-supported | unmatched |
|---|---|---|---|---|---|
| **PortMem** | **118.6s** | (standard review) | **7/8** | **0** | 0 |
| Windowed agent loop | 377.4s (3.2×) | 196 | 3/8 | **2** | 3 |

At the scale the product is actually sold for — corpora that cannot be hand-fed
to a model — the naive agent inverts: 3.2× slower AND it approved 2 planted
errors while failing to locate 3 more. Combined with the single-document tables
above, the honest composite is: an agent loop wins on speed for one in-context
document; PortMem wins on both speed and safety once the corpus exceeds the
window. Caveats: n=1 memo, 8 seeds, no repeats yet (single-document cells have
k=3; this cell should get repeats before any public speed wording returns —
founder decision either way). Artifacts: `eval/dogfood/corpus_scale/`.