# Multi-domain dogfood — review (A+B) beyond the Ambipar corpus

**Date:** 2026-07-09 · **Harness:** `product1/dogfood_domain.py` · **Judge:** Claude Haiku 4.5
(prod parity, `config/routing.haiku.yaml`) · **Embeddings:** Qwen3-Embedding-8B (4096d) ·
**Store:** local pgvector (`portmem-engine-pg`), source-scoped reviews.

Answers the review-panel ask: precision/stability evidence on document domains other than the
Ambipar indenture, using only public documents. Protocol per domain:

1. Ingest the public rules document (chunk+embed, own `source_id`).
2. Haiku writes a ~20-fact analytical memo FROM THE FULL DOCUMENT (the customer side, honest).
3. Haiku seeds K declared value errors into a corrupted copy (ground truth known).
4. `PortMem.review()` on the clean memo (A+B) and corrupted memo (A + seeded scoring).

## Results

| Domain | Corpus | Clean memo (false-alarm test) | B completeness | Corrupted memo (catch test) |
|---|---|---|---|---|
| FDA ELIQUIS label (openFDA) | 60KB, 43 chunks | score **1.0** — 21 sup / **0 contra** / 4 unc | 0 provisions (see note) | **4/4 in-review errors caught**, 0 false-supported |
| Apple 424B2 notes prospectus (EDGAR) | 160KB, 114 chunks | score **1.0** — 25 sup / **0 contra** / 0 unc | 27/30 covered, 3 omissions | **3/3 in-review errors caught**, 0 false-supported |
| EU AI Act Ch. I–III (EUR-Lex) | 145KB, 102 chunks | score **1.0** — 24 sup / **0 contra** / 1 unc | 25/30 covered, 1 material omission | **3/3 in-review errors caught**, 0 false-supported |

**Headline:** across 75 reviewed claims on three honestly-written memos in three new domains,
**zero false contradictions** (score 1.0 everywhere). Across all seeded value errors that
entered the review, **10/10 flagged** (contradicted or uncertain) and **0 came back
supported** — the unsafe failure mode did not occur once. Review wall-clock ~150s per pass.

## Caveats (read before quoting)

- **Claim-extraction sampling:** 11 of 21 seeded errors landed in memo sentences the extractor
  did not select at `max_claims=25` (the memos state 30+ facts). Those errors were never
  reviewed — not caught, not missed. A full-audit configuration should size `max_claims` to
  the memo. The clean 10/10 catch rate applies to errors that entered the review.
- **B is provision-shaped:** completeness extraction found 0 provisions in the FDA label
  (labels have no shall/must clause structure) and returned `completeness: null` — failing
  safe rather than inventing a score. B currently fits contracts/regulations (Apple, EU Act,
  indentures), not labeling. Known genre limitation.
- **One run per cell** (temperature 0 review, but B and generation are LLM-sampled);
  n per domain is one memo, ~25 claims. Directional evidence, not a benchmark.
- Uncertain verdicts on clean memos (4 FDA / 1 EU) are the calibrated overstatement flag
  doing its job on genuinely compressed claims, and cost nothing: they route to human review.

## Artifacts

`eval/dogfood/<domain>/`: `report_clean.txt`, `report_corrupted.txt`, `errors.json`
(declared ground truth), `review_clean.json`, `review_corrupted.json` (with
`seeded_scoring` mapping each error to its verdict). Corpora in `eval/dogfood/corpora/`.
Relation to prior evidence: complements the Ambipar per-source/adversarial line
(false-contradiction hunting) and the judge-tier logprob study (`logprob_judge_eval_*`).

## Re-run with sized extraction (2026-07-19, `--max-claims 45`)

Purpose: remove the extraction-sampling cap (the "10/21" asterisk). Same corpora, same protocol.

| Domain | Clean memo | Corrupted memo (seeded errors) |
|---|---|---|
| FDA ELIQUIS label | score 1.0 — 45 sup / **0 contra** | 4/7 caught, 0 wrongly approved, 3 unmatched |
| Apple 424B2 | score 1.0 — 45 sup / **0 contra** | 6/8 caught, 0 wrongly approved, 2 unmatched |
| EU AI Act Ch. I-III | score 1.0 — 34 sup / 11 unc / **0 contra** | 1/6 caught, **1 wrongly approved**, 4 unmatched (re-scored 2026-08-08, corrected matcher) |

**Combined across both runs (re-scored 2026-08-08 with the corrected diff-token
matcher — the original scorer matched any number in the corrupted phrase, degenerated to
single digits on two EU seeds, and credited an unrelated contradicted claim as their
catch):** clean memos still total **0 false contradictions across ~210 reviewed claims**
(unaffected — that number never depended on the matcher). End-to-end seeded catch is
**11/21** (not the previously published 13/21): FDA 4/7, Apple 6/8, EU 1/6, with 9 seeds
unmatched (never extracted as claims, so never reviewed). Of the 12 seeded errors that
actually entered review, 11 were flagged and **1 was wrongly approved** (the
count-aggregation miss below) — not the previously published "23/24", whose denominator
wrongly counted seeds that never entered review.

**The miss, in full (this is the honesty discipline working):** the seeded error changed "Eight
categories of AI practices are prohibited" to "Nine"; the extractor picked the claim up
verbatim and the judge marked it supported. Root cause: a **count-aggregation claim** — the
correct count (eight, Art. 5(1)(a)-(h)) spans multiple retrieved chunks, and no single passage
contradicts "nine." This is a known-hard claim class, now documented with a concrete production
example. Candidate mitigation: detect count/enumeration claims and route them to the
full-source judge (the fallback already exists for unresolved verdicts; extending its trigger
to count-type claims is scoped product work). Until then, the safe phrasing is: "one
count-aggregation error was wrongly approved out of the 12 seeded errors that entered review
across two runs; zero value-substitution errors (dates, rates, amounts, entities) were ever
approved."

## Count-aggregation guard — mitigation SHIPPED + guarded re-run (2026-07-20)

The scoped mitigation above is now implemented (`portmem/verify.py: is_count_claim,
count_claim_suffix`; `engine_binding.py` count-aggregation guard; `tests/test_count_guard.py`).
Mechanics: a SUPPORTED verdict on a count/enumeration claim is never trusted as-is. When the
complete source fits the full-text budget, the claim is re-judged with an ENUMERATE-then-count
instruction (list every item of the counted class with its passage id, then compare totals);
when it cannot be confirmed — or the full text is unavailable — the verdict is capped at
`uncertain` with `reason_code=count_claim_unverified` and routed to a human. The guard can
confirm (with itemized evidence), refute (via the refute-verified adoption path), or downgrade;
it can never mint a new `supported`, so the frozen judge calibration is untouched.
`count_guard=False` restores legacy behavior.

**Detector sweep (offline, all 270 historical dogfood claims):** fires on 3 (1.1%) — the seeded
"Nine categories" miss itself, plus "one of four conditions" (Art. 6(3) — genuinely a count
assertion; checking it is correct behavior). Zero firings on the FDA and Apple corpora: no
collateral on value-shaped claims (dates, rates, amounts, ratios, identifiers).

**Guarded EU AI Act re-run (same protocol, --max-claims 45):**

| Memo | Result |
|---|---|
| Clean | score 1.0 — 44 sup / **0 contra** / 1 unc; "Eight categories" and "one of four conditions" both CONFIRMED by enumerate-and-count review (conf 0.98-0.99, itemized evidence note) |
| Corrupted | **0 false-supported across all 6 seeds** (was 1 wrongly approved); 2/6 caught, 4 unmatched (re-scored 2026-08-08, corrected matcher; extraction sampling); the "Nine categories" seed lands `uncertain`, flagged for human review |

**Honest attribution:** in this live run the passage judge did not confidently support the
"Nine categories" seed in the first place (run-to-run extraction/retrieval variance); it was
declined by the pre-existing full-source fallback. The guard's cap path — what happens when the
passage judge DOES wrongly support a count claim, as in the 07-19 run — is proven
deterministically by `test_count_guard.py::test_supported_count_claim_capped_to_uncertain`,
which replays that exact scenario (passage judge says supported/0.95 → capped to `uncertain`).
The guard's confirm path is proven live (3 firings above).

**Safe phrasing now:** "count/enumeration claims are never auto-certified: a supported count is
re-verified by an enumerate-and-count review against the complete source, and capped at
'uncertain' for human review when that cannot confirm the total. In the guarded re-run, zero
seeded errors were approved and correct counts were confirmed with itemized evidence." Residual
limits (say when asked): detector recall is regex-bounded (unusual phrasings can slip it);
adjacent derived-truth classes (sums, date arithmetic, comparatives) are not yet routed; the
enumerating judge is itself an LLM — its output is auditable (itemized quotes), not infallible.

## Table_Claims — seeded TABLE-claim run (2026-08-08, live judge, prod parity)

The pending run from the tables batch (PR #70): synthetic rate-schedule corpus
(`corpora/table_rate_schedule.md`) whose load-bearing facts live in two markdown tables;
4 seeded errors, one per table failure mode (wrong cell, wrong COLUMN lookup, wrong ROW
lookup, wrong derived total). Product ingest path (table-aware chunker), Haiku 4.5 judge.

| Extraction | Clean memo (false-alarm test) | Corrupted memo (catch test) |
|---|---|---|
| max_claims=20 | score 0.947 — 18 sup / **1 contra** / 1 unc | 3/4 caught, 0 false-supported, 1 unmatched (total-sum claim not extracted) — *no retained artifact; the sized run's JSONs replaced this run's* |
| **max_claims=45 (sized)** | **score 1.0 — 31 sup / 0 contra / 0 unc**; B 22/23 | **4/4 caught, 0 false-supported, 0 unmatched** |

**Quotable:** at sized extraction, all 4 seeded table errors were flagged (contradicted or
uncertain), none came back supported, and the clean memo produced zero false contradictions.

Caveat that must travel with the number: at the undersized max_claims=20 run the extractor
merged the 3-row amortization schedule into one aggregate claim and the judge misread the
"Outstanding after payment" column against it — 1 false contradiction. Aggregated multi-row
claims are the residual risk; sized extraction removes it (same lesson as the 2026-07-19
sized-extraction re-run above). One run per cell; synthetic corpus; directional.
Route-token distribution (clean run, per its README requirement): all 31 claims resolved
via full-source review against the controlling instrument (source notes: "settled by
full-source review"); no `table_guard:capped` fired, i.e. the table-aware chunker delivered
tables with their headers and the guard never had to cap. Artifacts:
`eval/dogfood/Table_Claims/review_{clean,corrupted}.json`.

**Scoring rule (2026-08-08, applies to every table above):** a seed matches a claim only if
the claim carries the original→corrupted DIFF token on a word boundary with vocabulary
overlap; `unmatched` = the corrupted statement was never extracted, excluded from both
numerator and denominator of any "entered review" rate; a mixed supported+uncertain verdict
set is not a false-supported because the flagged verdict routes to human review (each seed's
matched claims + verdicts are stored in `seeded_scoring` for audit). All stored runs were
re-scored offline under this rule; corrected numbers replaced the originals above.

## Census-era repeats with error bars (2026-08-09, corrected scorer, k=3 per cell)

The eval audit's dominant limitation was n=1 per cell with high extraction variance.
With census extraction (now the review default) each corrupted-memo cell was run
3x (`eval/error_bars.py`; per-run JSONs in `<domain>/repeats/`):

| Cell | Run 1 | Run 2 | Run 3 | Variance | False-supported |
|---|---|---|---|---|---|
| Table_Claims (4 seeds) | 4/4 | 4/4 | 4/4 | none | 0 |
| EU AI Act (6 seeds) | 5/6 | 5/6 | 5/6 | none | 0 |
| FDA ELIQUIS (7 seeds) | 6/7 | 6/7 | 6/7 | none | 0 |

Two conclusions the earlier single-run tables could not support: census extraction
did not just RAISE catch (EU 1-2/6 → 5/6, FDA 4/7 → 6/7 vs the single-pass runs),
it made the rate REPRODUCIBLE — the run-to-run extraction variance that moved
single-pass results by 15-30 points is gone at k=3. And the safety invariant held
across all 9 runs / 51 seed-checks: zero planted errors approved. Each cell's
remaining miss is the same seed every time (a stable extraction gap, not noise) —
per-seed detail in the repeat JSONs identifies them for the next extraction
iteration. Still synthetic corpora and self-seeded errors; the human-labelled
gold sets remain founder-side.

## Whole-pipeline campaign (2026-08-09, pre-registered protocol — eval/PROTOCOL.md)

4 domains × 3 memos, 6 seeded errors each at k=3 repeats, plus a seeded-OMISSION
variant (3 provisions deleted per memo) for completeness recall. 255 seed-checks,
1,475 clean claims judged. Bootstrap 95% CIs over seeds. Raw runs:
`eval/dogfood/campaign_2026-08/` + the public evidence portal.

| Domain | Extraction recall | Catch (of extracted) | False-supported | Clean false alarms | Omission recall (B) |
|---|---|---|---|---|---|
| FDA ELIQUIS | 0.90 [0.87, 0.95] | 0.89 [0.85, 0.95] | **5 instances** | 1/459 | B unavailable (0 provisions) |
| Apple 424B2 | 0.85 [0.74, 0.94] | 1.00 | 0 | 0/284 | B unavailable (0 provisions) |
| EU AI Act | 0.79 [0.78, 0.81] | 0.98 [0.95, 1.00] | 1 | 0/600 | **0/9** |
| Table_Claims | 1.00 | 1.00 | 0 | 0/132 | 0.78 [0.67, 1.00] |

**The campaign RETIRES the "zero planted errors ever approved" claim.** At n=255 the
measured false-supported rate is **6/255 ≈ 2.4%** (4 distinct seeds). Per-seed audit:
- 2 seed-quality artifacts: the corrupted value coincides with a REAL corpus value in a
  nearby context (EU "48 hours" exists as a genuine provision; FDA "88%" sits next to
  the label's real 87%/92%), so "support" reflects value collision, not blindness.
- 2 genuine judge failures: FDA "71% vs 69% age ≥65" (corpus says 69, no 71 anywhere)
  and FDA "6-22× exposure" (approved 3/3 runs; the corpus's real ranges are 1.4-5× —
  and the memo generator's own "original 4-19×" was ALSO corpus-false, i.e. the clean
  memo carried a hallucinated value the clean-run judge did not catch either).
The safe public phrasing is now a measured RATE with CIs, not a zero.

**Other findings, stated plainly:**
- Clean false alarms: 1/1,475 (~0.07%) — one contradiction on a clean FDA memo.
- Extraction recall 0.79-1.00: census closes most of the gap; EU's dense-list genre
  remains hardest (stable, tight CI).
- Completeness (B) is genre-limited: 0 provisions extracted on FDA (known) AND on
  Apple 424B2 (NEW — the 2026-07 run found 27/30 there; regression or memo-scope
  sensitivity, under investigation), and on EU it extracted 30 provisions yet flagged
  0/9 seeded deletions — B's provision list does not track the memo's material
  content closely enough to catch targeted omissions. **B needs redesign before any
  completeness recall claim.** Table_Claims (0.78) is the only domain where the
  omission promise currently holds.

## Campaign re-measurement (2026-08-09 pm) — SUPERSEDES the table above

The B-redesign session re-ran the affected campaign cells: the Apple domain was
re-run end-to-end after the source-id typo fix (its original runs had silently
verified against the WHOLE corpus, not the scoped source — invalid), and FDA/EU
memo cells were regenerated in the same pass. The table above is kept visible per
the corrections policy; the numbers below are the current canonical ones
(`eval/dogfood/campaign_2026-08/summary.json`).

| Domain | Extraction recall | Catch (of extracted) | False-supported | Clean false alarms | Omission recall (B, redesigned) |
|---|---|---|---|---|---|
| FDA ELIQUIS | 0.90 [0.87, 0.95] | 0.89 [0.85, 0.95] | 5 | 1/459 | 4/9 (0.44) |
| Apple 424B2 | 0.86 [0.78, 0.93] | 0.85 [0.78, 0.92] | **9** | 1/287 | 6/9 (0.67) |
| EU AI Act | 0.79 [0.78, 0.81] | 0.98 [0.95, 1.00] | 1 | 0/600 | 2/9 (0.22) |
| Table_Claims | 1.00 | 1.00 | 0 | 0/132 | 7/9 (0.78) |

Pooled: **false-supported 15/255 (~5.9%)** — up from the earlier 6/255 because the
properly source-scoped Apple runs no longer had the whole corpus to draw on and
approved 9 seeded corruptions (distinct seeds: the 2030-Notes principal/coupon
sentence, the 0.250% underwriting discount, the co-manager count, the 2032-Notes
coupon). Catch-or-flag of extracted seeds 208/224 (92.9%). Clean false alarms
2/1,478 (~0.14%). Omission recall (B, post-redesign) 19/36 (0.53). The earlier
prose quoting 6/255 and 1/1,475 is superseded; per-seed detail for every
false-supported case is in the run JSONs (grep `"false_supported": true`).

## Completeness (B) redesign + retest (2026-08-09, post-campaign)

The campaign's B findings drove a redesign, tested against the campaign's own seeded
deletions until working: genre-adaptive derivation (labels/regulations, not just
contracts), NO early union cap with round-robin merging across document regions,
rule-confirm against each provision's own derivation context (not a sparse sample),
a sharpened own-terms coverage judge (topical adjacency no longer credits — a
prohibition is not its authorization procedure), quote-adjudicated refute verdicts,
and a ranked top-10 flag budget with the remainder disclosed as a count. Also fixed:
the campaign harness's Apple source-id typo (named sources now FAIL LOUD on zero
chunks), and scorer v2 for omission matching (collective distinctive-term coverage).

| Domain | Seeded-deletion recall (was) | Clean-memo flags (budgeted) |
|---|---|---|
| FDA ELIQUIS | 4/9 (was structurally unavailable) | 6 |
| Apple 424B2 | 6/9 (was unavailable — harness typo + cap) | 10 (+63 disclosed as count) |
| EU AI Act | 2/9 (was 0/9) | 0 |
| Table_Claims | 7/9 (unchanged) | 6 |

Overall seeded recall 19/36 (0.53) from ~0. Honest residuals: EU's derived labels
still misalign with some deleted slices (adjacent-provision granularity); clean-memo
flags are LITERALLY-true uncovered material provisions — for short memos this is
correct behavior with a bounded review burden, not false positives in the A sense.
Run-to-run label stochasticity moves individual cells ±1. Raw runs in
campaign_2026-08/*/memo*/review_omitted.json (regenerated).