# Held-out campaign results (2026-08-09, protocol: eval/HELDOUT_PROTOCOL.md)

First fully blind evaluation: four corpora the pipeline had never seen, memos
generated once, no cell re-run for its score (two documented harness amendments,
both at the generation step, in the protocol). 243 seeded-error checks, 1,327
clean claims, 36 seeded deletions. Published as measured.

## Per-stage results (point estimate [95% bootstrap CI])

| Domain | 1. Extraction recall | 2. Catch of extracted | 2b. False-supported | 3. Clean false alarms | 4. Completeness recall |
|---|---|---|---|---|---|
| FDA XARELTO label | 1.00 | 0.95 [0.91, 0.98] | 3/63 | 0/258 | 1/9 (0.11) |
| Oracle 424B2 notes | 0.94 [0.89, 1.00] | 0.90 [0.85, 0.96] | 5/54 | 0/335 | 3/9 (0.33) |
| EU GDPR Ch I–V | 0.70 [0.56, 0.85] | 1.00 | 0/54 | 1/569 | 0/9 (0.00) |
| Table premium schedule | 0.88 [0.81, 0.94] | 1.00 | 0/72 | 0/165 | 4/9 (0.44) |
| **Pooled** | — | — | **8/243 (3.3%)** | **1/1,327 (0.075%)** | **8/36 (0.22)** |

## Held-out vs development set, stated plainly

| Metric | Development (tuned-on) | Held-out (blind) | Verdict |
|---|---|---|---|
| False-supported rate | 15/255 (5.9%) | 8/243 (3.3%) | Generalizes (no degradation) |
| Clean false alarms | 2/1,478 (0.14%) | 1/1,327 (0.075%) | Generalizes |
| Extraction recall range | 0.79–1.00 | 0.70–1.00 | Generalizes; dense-list regulation text is the weak stage on BOTH sets (GDPR 0.70 mirrors EU AI Act 0.79) |
| Catch of extracted | 0.85–1.00 | 0.90–1.00 | Generalizes |
| Completeness (B) recall | 19/36 (0.53) | 8/36 (0.22) | **Does NOT fully generalize.** The redesign was iterated against the development set's seeded deletions, and roughly half its measured gain was fitting to those memos. GDPR: 0/9. |

Two honest conclusions:

1. **The verification stages (extract → judge → clean specificity) hold up
   blind.** Every number a customer relies on for "was a stated fact checked and
   was the verdict safe" is as good or better on documents we never touched.
   Every seeded error that was extracted and then missed is in the raw JSONs.
2. **Completeness does not yet earn a recall claim.** 0.53 on the development
   set was partly fit; the blind number is 0.22 pooled and 0.00 on dense
   regulation text. B remains a disclosure surface (it finds real omissions and
   discloses its own truncation) — it is not a guarantee of finding what a memo
   omits, and no public wording may imply otherwise until a blind number
   supports it.

## Per-seed detail

Raw run JSONs: `eval/heldout/campaign/<domain>/memo<i>/` (repo + evidence
portal, auto-published). False-supported instances: 3 on XARELTO (1 seed ×3
repeats — same stable seed each run), 5 on Oracle (2 seeds). Grep
`"false_supported": true`.