{"data":{"kind":"file","path":"README.md","version_id":"fo2bvq02otr3y318re0zuqv6","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":14305,"modified_at":"2026-11-08T03:45:29.262000","content_hash":"8d93ea05905620cc30cd7ca206ce7d7671ad6eee2d442c14d04a0dac5e4bb436"},"entries":[],"content":"# Can a small open model match a frontier model at reading business documents?\n\n**Not yet, and the gap is smaller than it first looks.** Opus 5.5 scores 0.9121\nagainst 0.7985 for a 30B and 0.7272 for a 9B on OCR text, but roughly a third of\nthe frontier-over-30B gap turned out to be my prompt being ambiguous rather than\nthe model being weak. I measured that share, pre-registered what fixing it\nshould recover, fixed one sentence of the prompt, and re-ran: it recovered 83%\nof the predicted ceiling for $0.20.\n\nA single-turn RL environment for structured field extraction from real\nreceipts, on `verifiers` v1, with a verifiable reward and a verifier tested\nagainst ground truth, seven corruptions and three gaming strategies before any\nmodel ran. 6,000 scored rollouts, 0 failures.\n\nFull study, pre-registration and friction log:\n[github.com/maker-sean/rl-env-headroom-study](https://github.com/maker-sean/rl-env-headroom-study)\n\n```bash\nprime env install maker-sean/doc-extraction-study\nvf-run doc-extraction-study -m qwen/qwen3.5-9b -n 5 -r 1\n```\n\n## The task\n\nGiven one receipt, as a photograph or as OCR text, return a fixed JSON schema:\nline items (name, quantity, unit price, price) plus subtotal, tax, service\ncharge, discount and total. Fields the receipt does not show must be omitted,\nnot guessed.\n\nBoth conditions run over the same documents with the same prompt, so the delta\nbetween them isolates reading from reasoning and formatting, which a single\naccuracy number cannot do.\n\n## Results\n\nThree tiers, two conditions, 100 held-out test documents, 10 samples each at\nT=0.7. Both conditions were run for every tier. Every table is generated from\nthe files in `results/` by `scripts/21_regenerate_tables.py`; nothing is typed\nby hand, and `--check` fails if any number drifts.\n\n<!-- BEGIN GENERATED: regenerate_tables.py -->\n\n### Mean reward, and the effect of input modality\n\n| tier | model | OCR text | page image | image minus OCR | 95% CI |\n|---|---|---|---|---|---|\n| Qwen3.5-9B | `qwen/qwen3.5-9b` | 0.7272 | 0.7169 | +0.0103 | [-0.0192, +0.0416] |\n| Qwen3-VL-30B-A3B | `qwen/qwen3-vl-30b-a3b-instruct` | 0.7985 | 0.8203 | -0.0218 | [-0.0475, +0.0044] |\n| Opus 5.5 | `anthropic/claude-opus-5.5` | 0.9121 | 0.9208 | -0.0087 | [-0.0287, +0.0110] |\n\nEvery interval spans zero. Minimum detectable effect by tier: 0.0304 / 0.0259 / 0.0198. Paired bootstrap over the same 100 documents, 10,000 resamples, seed 20261106.\n\n### Scale gaps, which do not span zero\n\n| gap | condition | gap | 95% CI |\n|---|---|---|---|\n| frontier-mid | ocr_text | +0.1136 | [+0.0825, +0.1459] |\n| mid-small | ocr_text | +0.0713 | [+0.0292, +0.1163] |\n| frontier-mid | image | +0.1006 | [+0.0741, +0.1292] |\n| mid-small | image | +0.1034 | [+0.0564, +0.1539] |\n\nTier ordering holds under 6 of 6 reward rubrics, so it is not an artifact of the weights chosen.\n\n### Document exact match and pass@k\n\nA document counts only when every field is right. pass@k is over the 10 samples drawn per document.\n\n| tier | condition | n | exact match (pass@1) | pass@3 | pass@5 | pass@10 |\n|---|---|---|---|---|---|---|\n| Qwen3.5-9B | `ocr_text` | 1,000 | 0.300 | 0.390 | 0.433 | 0.500 |\n| Qwen3.5-9B | `image` | 1,000 | 0.276 | 0.363 | 0.402 | 0.450 |\n| Qwen3-VL-30B-A3B | `ocr_text` | 1,000 | 0.360 | 0.386 | 0.393 | 0.400 |\n| Qwen3-VL-30B-A3B | `image` | 1,000 | 0.413 | 0.454 | 0.468 | 0.490 |\n| Opus 5.5 | `ocr_text` | 1,000 | 0.674 | 0.684 | 0.688 | 0.690 |\n| Opus 5.5 | `image` | 1,000 | 0.664 | 0.675 | 0.680 | 0.690 |\n\n### How much of the gap was my specification, not the model\n\nRe-scoring saved rollouts under one named change at a time. The ablations are deliberately generous, so these are upper bounds.\n\n| gap on OCR text | measured | upper bound after spec fixes | share specification |\n|---|---|---|---|\n| frontier over mid | 0.1136 | 0.0700 | 38% |\n| frontier over small | 0.1849 | 0.1465 | 21% |\n\nForgiving an invented `subtotal` is worth +0.0379 to the mid tier and +0.0001 to the frontier. A fix worth something to the smaller model and nothing measurable to the frontier is a fix to the task, not to the model.\n\n### The fix, pre-registered and re-run on `mid|ocr_text`\n\n| quantity | before | after | change | 95% CI |\n|---|---|---|---|---|\n| mean reward | 0.7985 | 0.8301 | +0.0316 | [+0.0176, +0.0460] |\n| doc exact match | 0.3600 | 0.4970 | +0.1370 | [+0.0700, +0.2080] |\n| invented `subtotal` | 282 | 53 | -229 | not bootstrapped |\n| missed `subtotal` | 0 | 0 | +0 | not bootstrapped |\n\nOne sentence added to the prompt, prompt version `v1.1.0-subtotal`, cost $0.2024. Missed subtotals did not rise, so there was no precision-recall trade: that is the evidence this was a specification gap and not a reading one.\n\n### What remains, and it is capability\n\nCounts are **rollouts of 1,000 per cell** carrying at least one such error, on OCR text.\n\n| error | Qwen3.5-9B | Qwen3-VL-30B-A3B | Opus 5.5 |\n|---|---|---|---|\n| value wrong by 1000x (locale separator) | 141 | 4 | 0 |\n| missed `tax` | 75 | 62 | 0 |\n| price wrong on a matched item | 155 | 99 | 6 |\n| item name mismatch | 311 | 233 | 151 |\n| invented `subtotal` | 254 | 282 | 21 |\n| unparseable reply | 0 | 0 | 0 |\n\nThe mirror image: where the prompt *is* explicit, that `price` means the line total rather than the per-unit amount, **0 rollouts of 6,000 confused the two**, with 135 opportunities in the richest cell across 18 observable items. Silent prompt and the models diverge; explicit prompt and even the 9B complies.\n\n### Pre-registered predictions, scored\n\n| author | predictions | hit | partial | miss |\n|---|---|---|---|---|\n| Claude | 7 | 1 | 2 | 4 |\n| Sean | 5 | 3 | 1 | 1 |\n\n### Cost, reliability, and the verifier's own gates\n\n- Claim run: 6,000 rollouts, 0 failures, $20.58.\n- All runs in this repo: 7,424 rollouts across 21 logged runs, $23.08 total, itemised in [`results/spend.csv`](results/spend.csv).\n- Verifier gate: ground truth scores 1.0000 on all 100 validation documents (min 1.0000, max 1.0000).\n- Every corruption and gaming strategy scores below ground truth; see [`results/verifier_selftest.json`](results/verifier_selftest.json).\n\n<!-- END GENERATED -->\n\n## Specification, not capability\n\nOn receipts that print only a total, the 30B emits a `subtotal` equal to that\ntotal in 282 of 1,000 rollouts. My prompt never said whether to report a field\nthe document does not print, and the frontier model happened to resolve that\nambiguity the way CORD's annotators did. Re-scoring saved rollouts while\nforgiving that one error is worth +0.0379 to the 30B and +0.0001 to the\nfrontier, and a fix worth something to the smaller model and nothing measurable\nto the larger one is a fix to the task rather than to the model. That is\ninstruction-following wearing an extraction label, and the asymmetry is how you\ndetect it. Adding one sentence to the prompt moved mean reward +0.0316\n[+0.0176, +0.0460] and document exact match from 0.360 to 0.497, without\nsuppressing a single subtotal that *was* printed, so there was no\nprecision-recall trade. The original prompt is still available as\n`--env.taskset.prompt_variant v1.0.0` so the ablation stays reproducible.\n\n## The locale failure, which is a training target\n\nThe clean capability gap. CORD receipts are Indonesian and print `60.000` for\nsixty thousand; the 9B emits `60.0`. Per 1,000 rollouts on OCR text that error\nappears **141 times for the 9B, 4 times for the 30B, and 0 times for Opus\n5.5**. No ablation recovers any of it, so it is not a specification problem, and\na wrong item name is a data-quality defect while a value wrong by 1000x is a\nfinancial one.\n\nIt is a good RL target for three reasons: the correct answer is a deterministic\ninteger, the reward is already per-field so the signal is not diluted by the\nrest of the document, and the 800-document `train` split is untouched by this\nstudy. **No training was run here.** That claim is an argument from the reward's\nshape, not a measurement.\n\n## The harness default, and why this package overrides it\n\nWorth knowing before you run any toolless taskset on `verifiers` v1.\n`default_harness_id()` returns `\"bash\"` unless the taskset's own package\nexports a `Harness` subclass. For a single-turn extraction task the `bash`\nharness is actively wrong: it prepends \"You are a coding agent. You have access\nto a bash tool...\" ahead of the task's system prompt and hands the model tools.\nIn this study's first pilot the 30B made a tool call, got an empty result,\nanswered in prose, and scored **0.0000 on all 60 rollouts**, which reads as a\nmodel failure rather than a misconfiguration. That pilot cost $1.25 and was\ndiscarded.\n\nThis package exports `DocExtractionHarness`, a `NullHarness` subclass, so the\ncorrect harness is the default with no flag to remember. Verified on the\npublished package: `default_harness_id(\"doc-extraction-study\")` returns\n`doc-extraction-study`, not `bash`, and the command at the top of this README\nlogs `harness=doc-extraction-study`, one turn, `stop=agent_completed`, with a\nnonzero reward.\n\nIf you build your own toolless taskset, export a `NullHarness` subclass or pass\n`--env.agent.harness.id null`. A 0.0000 that looks like a weak model is worth\none check of the request payload.\n\n## Data provenance and license\n\n[CORD v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2), 1,000 real\nphotographed receipts, **CC-BY-4.0**, verified from two independent sources with\nthe license text archived in `provenance/DATASET_LICENSE.md`. CORD is not\nredistributed here; the loader fetches it.\n\nCORD contains no merchant name and no date, as its publishers removed those\nbefore release, so they are out of scope. **No synthetic fields are injected**,\nso every scored target is real ground truth. Receipts are predominantly\nIndonesian: these are document-understanding results, not a claim about any\nparticular document mix.\n\nCode in this package is MIT. If you use CORD, cite it as its authors ask:\n\n> Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., Lee, H. *CORD: A\n> Consolidated Receipt Dataset for Post-OCR Parsing.* Document Intelligence\n> Workshop at NeurIPS, 2019.\n\n## Rubric\n\nA weighted sum, as verifiers v1 requires. Invalid JSON scores zero on every\ncomponent.\n\n| component | weight | what it measures |\n|---|---|---|\n| `scalar_f1` | 0.45 | F1 over (field, value) pairs. Absent is a value, so inventing a field is a false positive |\n| `line_item_f1` | 0.35 | F1 over line items; a true positive needs both the name and the price |\n| `grounding` | 0.15 | Fraction of emitted numbers that appear in the document, diluted if the model emits more values than the document contains |\n| `schema_valid` | 0.05 | Produced well-formed JSON matching the schema |\n\n**Arithmetic consistency is measured but never rewarded.** CORD's own ground\ntruth satisfies the line-item sum on only 91% of evaluable documents, because\nIndonesian receipts use tax-inclusive pricing with rounding. Rewarding\nconsistency would penalize correct extractions.\n\nBecause the weights are a choice, `results/weight_sensitivity.json` re-scores\nevery rollout under six rubrics and reports which conclusions survive. Tier\nordering holds under all six. The modality null holds under all six at the\nfrontier and breaks under two at the 30B.\n\n## Limitations\n\n- **Evaluation only.** No RL training was run.\n- **100 test documents.** Minimum detectable effect is roughly 0.02 to 0.03 by\n  tier, so \"no modality effect\" means no effect above that.\n- **The small-to-mid gap confounds scale with model family**, because the\n  intended small tier was not servable on the gateway. It is reported as a tier\n  gap, never as a scaling law.\n- **The headroom ablations are upper bounds by construction.** The\n  pre-registered prompt re-run exists to test one of them against reality.\n- **Exact match carries a dataset ceiling.** About 9 of 100 documents print more\n  item lines than CORD annotates, so the frontier's 0.674 is not 0.326 from\n  solved.\n- **CORD has been public since 2019**, so the frontier score could be partly\n  memorization. Exact match at 0.674 and disagreement with CORD's literal item\n  names argue against it; the perturbation test was not run.\n- **The label audit is 20 of 100 documents**, hand-checked against the receipt\n  images: 19 clean, 0 label bugs. That puts the 95% upper bound on the label\n  error rate near 14%, which is what 20 documents buy.\n\n## What ships, and where\n\n`prime env install` installs the wheel, which is the environment itself:\n\n| path | what |\n|---|---|\n| `src/doc_extraction_study/taskset.py` | the verifiers v1 taskset, both conditions |\n| `src/doc_extraction_study/scoring.py` | the verifier: pure, testable scoring |\n| `src/doc_extraction_study/normalize.py` | amount, quantity, name normalization |\n| `src/doc_extraction_study/cord.py` | CORD to a canonical record, OCR reconstruction |\n| `src/doc_extraction_study/prompts.py` | the fixed prompt, versioned, plus the one variant |\n| `src/doc_extraction_study/harness.py` | the `NullHarness` subclass described above |\n\n`prime env pull maker-sean/doc-extraction-study` additionally gives you the\nwhole study: `results/` (every figure quoted anywhere traces here, including\n`spend.csv`), `scripts/` (the probes, the verifier self-test, the eval runner\nand every scorer), `provenance/` (dataset license, model catalog snapshot, the\nlabel audit), `docs/HEADROOM.md` (the ablations and what each one bounds),\n`docs/DATA_NOTES.md`, `docs/RL_FEASIBILITY.md`, `PREREGISTRATION.md`, and\n`audit_review/` (the page used to hand-audit the receipts).\n\n## Reproduce the full study\n\nFrom a pulled copy, with `uv sync && uv pip install -e .`:\n\n```bash\nuv run pytest                                      # 101 unit tests\nuv run python scripts/04_verifier_selftest.py      # the three gates, no spend\nuv run python scripts/21_regenerate_tables.py --check   # README vs results/\n```\n\nThe gold check, which runs no model:\n\n```bash\nuv run vf-validate doc-extraction-study --only-gold --taskset.limit 5 \\\n  --runtime.type subprocess\n```\n\nRe-running the models costs roughly $20 at the prices recorded in\n`provenance/`:\n\n```bash\nuv run python scripts/09_run_eval.py --split test --samples 10\n```\n\nNote that `vf-eval` and `vf-init` appear in some tutorials and do not exist in\n`verifiers` 0.4.1. The commands above are the ones that work.\n","encoding":"utf-8","truncated":false,"total_bytes":14305},"status":null}