{"data":{"kind":"file","path":"README.md","version_id":"kqnyw05rp0m4wpsj15n6nnbp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":11363,"modified_at":"2026-10-12T05:42:26.704000","content_hash":"0671859f911a1fe539cb6ee805a2e607d12b903631b3696a736618e4daf2887a"},"entries":[],"content":"# Redtape: verifiable abstention for public-benefits determinations\n\n> ## Retraction (2026-09-21): the category/quantity finding was an artifact\n>\n> This environment's headline claim (that a frontier model notices a missing fact far more\n> reliably when its absence changes a **category** than when it changes a **quantity**,\n> measured at 0.396 against 0.050) **is retracted.** It does not hold on the corrected\n> corpus, on either model tested.\n>\n> **The reason is a design defect, not a measurement error.** The generator routes each\n> withheld fact to whichever stream can produce a given class, so **class and withheld fact\n> are entangled by construction**: 55% of the eligibility-flip class is a single fact\n> (`p1.employment_income`, 53 of 96), and that fact has **one task of 180** in the\n> indeterminate class. The comparison was never between \"category\" and \"quantity\". It was\n> between two different mixtures of facts. **No model run on this design can answer the\n> question**, and the original 0.396/0.050 is plausibly the same confound. Holding the fact\n> constant, the flip class does *worse* in five strata of six (pooled difference −0.153).\n>\n> A design that could test the claim needs each fact to appear in both classes in balanced\n> proportions. That is scoped in `docs/LIMITS.md` §44 and **not built**.\n>\n> **What the corrected corpus does support, across two labs' models**, both measured on the\n> same 1,194 tasks, the full split:\n>\n> | | Claude Opus 5 | GPT-5.6 Sol |\n> |---|---|---|\n> | abstention accuracy | 0.624 [0.577, 0.669] | **0.793** [0.752, 0.829] |\n> | exact-match (determinate) | 0.621 [0.587, 0.655] | 0.627 [0.592, 0.660] |\n> | pair-consistency | 0.636 (n=198) | 0.675 (n=200) |\n> | named the missing fact in exact form | **100%** (156/156) | 93.2% (220/236) |\n> | named a fact that was never withheld | **0%** | 7.6% (18) |\n>\n> GPT-5.6 Sol is better calibrated about when it cannot answer. Claude Opus 5 is better at\n> saying *which* fact is missing and never invents one. **The two abilities are\n> independent**: a benchmark measuring only format compliance would rank these models the\n> other way round.\n>\n> **A third model, Claude Opus 5.5** (1,200 of 1,200, same corpus and configuration), closed\n> most of the abstention gap: **0.745** [0.701, 0.785], up 0.121 on Opus 5 (paired 95% CI\n> [+0.084, +0.159]) and 0.048 below GPT-5.6 Sol (paired [−0.095, −0.001], McNemar p = 0.059;\n> the unpaired interval includes zero). That suggests the Opus 5 vs GPT-5.6 Sol difference was\n> mostly generational. **Whether a lab-level residual remains is unresolved.** It still names\n> the missing fact in exact form every time (187/187) and never invents one.\n>\n> Its exact-match fell to 0.563, and the drop is a **temporal rule-version error**: 89 SNAP\n> answers sit exactly $10 above the key because Opus 5.5 applies the FFY2026 maximum\n> allotments to FFY2025 months. Opus 5 made the same kind of error the other way round,\n> applying the pre-2025 $2,000 child tax credit where PL 119-21 sets $2,200 (exactly $200 short\n> per child, 54 times). The keys are right in both cases. A model that knows a rule's value but\n> not the dates it is in force is off by exactly the size of the rule change. It is the\n> clearest evidence so far for rulebooks versioned by date. No corrected exact-match figure is\n> claimed; the account is in `docs/LIMITS.md` §45.\n>\n> **This version ships the corrected corpus.** Every premise an answer key depends on is\n> stated in the case file (relationships and filing structure, weekly hours, Social\n> Security status, heating and cooling costs), and a test fails if that stops being true.\n> The v0.1.0 corpus defects (two-adult households keyed as married couples, undocumented\n> filers keyed as holding SSNs, students keyed at zero work hours) are recorded in\n> `docs/LIMITS.md` §35 and §36. **Every result measured on v0.1.0 is superseded. Please do not\n> cite it.**\n>\n> Nothing here is a PolicyEngine defect. The statutes were read (IRC §32(c)(1)(E) and §32(m),\n> §24(h)(7) as amended by PL 119-21, 7 CFR 273.5(b)); the engine matches them. Each wrong key\n> came from an input our own oracle left unset, so the engine supplied a default.\n>\n> The pre-registered decision rule, the result reported exactly as that rule dictates\n> (+0.126, CI [+0.004, +0.245], p = 0.056, on the full 1,200-task split the rule required),\n> and the disclosed reasons for departing from it are in `docs/LIMITS.md` §43.\n\n**Does the agent know when a required fact is missing?** Most benefit-calculation evals score\nwhether the number is right. This one scores whether the agent notices it *cannot* produce a\nnumber, and says so.\n\nEach task is a household narrative that withholds exactly one fact. Ground truth comes from\n[PolicyEngine](https://github.com/PolicyEngine/policyengine-us), a real microsimulation\nengine used for actual policy analysis, not from another model's opinion.\n\n## Installs in seconds: no microsimulation engine required\n\nThis is worth stating before anything else, because it is the difference between an\nenvironment people try and one they mean to try.\n\n**Evaluation never touches PolicyEngine.** Answer keys are computed once, at generation time,\nand baked into the split; nothing on the scoring path imports the engine. So installing this\nenvironment does not pull a microsimulation engine, its dependency tree, or its parameter\ndatabase. `policyengine-us` is an optional `[generate]` extra, needed only if you want to\nbuild a *new* split from a different seed or tax year.\n\nThe runtime dependencies are `verifiers` and `pydantic`. That is the whole list.\n\n```bash\nprime env install jakpotvin/redtape@latest\n```\n\n```python\nfrom redtape import T1Taskset\nfrom redtape.envs.t1_eligibility import T1Config\n\ntaskset = T1Taskset(T1Config())   # 1,200 tasks ship with the package\n```\n\nNo API key is needed to load the tasks or to score an answer. The installed package is the\ntaskset, oracle and scoring; the baselines and test suite ship in the environment *source*\n(`prime env pull`, or the GitHub repo) and also need no key.\n\nRun in a pulled copy with the `[dev,generate]` extras installed, the suite reports\n**384 passed, 41 skipped**. The copy is not a git repository and leaves out the repository's\n`.gitignore`, CI workflow, results files and README (this page takes its place), so the 38\nchecks on those skip and name what is missing; the other 3 skip in a clean checkout of the\nrepository too. Measured on 2026-10-12 in the tree that is uploaded. In version 0.1.7 the\nsuite did not pass in a pulled copy: `docs/LIMITS.md` §50.\n\n## What makes it different\n\n- **Deterministic ground truth, no LLM judge.** Answer keys trace to named PolicyEngine\n  variables. Rewards are computed, not rated, so the same response scores the same way\n  every time, enforced by a golden-master determinism test in CI.\n- **Abstention is scored in both directions.** Every task withholds one fact, labelled into\n  three classes: the fact decides the outcome (abstaining is correct), the fact is missing\n  but does *not* decide it (**answering** is correct), or nothing is withheld. That middle\n  class is what stops \"always abstain\" from winning: it scores 0.000.\n- **Paired tasks.** 200 pairs differ in exactly one attribute, half of which should change\n  the answer and half of which should not, so a model cannot score well by being uniformly\n  cautious or uniformly confident.\n\n## What the metrics do\n\nThree headlines, reported separately. The weighted composite exists but is deliberately\n**not** the headline, so retuning a weight cannot move a published number.\n\nCorrected corpus, full dev split. Each model row is that model's own run (exact-match n is\n776, 780 and 778 respectively, from the tasks each answered); every baseline covers all 780.\n\n| | exact-match | abstention (n=420) | pair-consistency |\n|---|---:|---:|---:|\n| **Claude Opus 5** | **0.621** | **0.624** | **0.636** (198 pairs) |\n| **Claude Opus 5.5** | **0.563** | **0.745** | **0.705** (200 pairs) |\n| **GPT-5.6 Sol** | **0.629** | **0.793** | **0.675** (200 pairs) |\n| always_abstain | 0.000 | 0.000 | 0.000 |\n| never_abstain | 0.194 | 0.340 | 0.480 |\n| always_eligible | 0.041 | 0.343 | 0.500 |\n| never_eligible | 0.123 | 0.083 | 0.085 |\n| rules_only | 0.194 | 0.376 | 0.480 |\n| *ceiling agent* | *1.000* | *1.000* | *1.000* |\n\nGPT-5.6 Sol's exact-match is 0.629 here, over the 778 determinate tasks its own run answered.\nThe notice above gives 0.627, computed on the 774 determinate tasks that both it and Claude\nOpus 5 answered, so the two models are compared on identical tasks. Both are correct for\ntheir task sets.\n\nNo trivial strategy exceeds 0.50 on any headline, and the ceiling agent proves every metric\nis actually achievable. Until it was written, nothing established that the abstention metric\nwas reachable at all. All six are scripted rather than model-driven, so they cost nothing and\nneed no API key, and they reproduce the numbers above exactly.\n\nOne caveat, since the point of this README is not to overstate the engine-free claim: the\n*baselines* do read the federal poverty line from PolicyEngine, so running them needs\n`pip install -e \".[dev,generate]\"` from the source tree. The engine-free property belongs to\nthe **evaluation path** (loading tasks, prompting, parsing and scoring), which is what an\ninstalled `prime env install` gives you, and which was verified in a clean environment with\n`policyengine_us` absent.\n\n## Superseded result (v0.1.0 corpus): retracted, do not cite\n\n**Kept for the record only. This is the finding the notice at the top retracts.** It was\nmeasured on the v0.1.0 corpus and does not hold on the corrected one. The current results\nare in the notice above.\n\nClaude Opus 5, 1,200 tasks, no tools:\n\n| the withheld fact would change… | correct abstention |\n|---|---:|\n| SNAP eligibility, a **category** | **0.396** (38 / 96) |\n| a benefit amount, a **quantity** | **0.050** (9 / 180) |\n\nAn eight-fold gap between two classes that differ in one respect: whether the missing fact\nmoves a yes/no or a number.\n\nIt is not incapacity at the task: exact-match is 0.514 against a 0.205 best baseline. And it\nis not blanket caution: where the missing fact does *not* decide the outcome, so answering\nis correct, it answers 95.1% of the time. It is discriminating, on the wrong axis.\n\n## Scope, and what is not validated\n\nOne model, one state (California), one tax year, one prompt. Abstention labels come from a\nperturbation sweep, which can prove a fact is deciding but cannot prove one is not.\nMedicaid is computed but deliberately **unscored**: no external validation was obtainable\nfor it. A held-out split exists, has never been evaluated against, and stays that way.\n\n**Read [`docs/LIMITS.md`](docs/LIMITS.md) before citing any number from this environment.**\nIt is 50 sections, written as the work happened rather than retrofitted, and several of them\nretract errors found in my own published results, including a schema bug that was\npenalising exactly the behaviour the benchmark exists to reward, and which sat visible in\nevery report for two days before it was read correctly.\n\n---\n\nSource, full README, and the raw results files:\n**https://github.com/JakPot42/redtape** · Apache-2.0 · 425 tests (the repository README\nsays what a run reports, and where that was measured)\n","encoding":"utf-8","truncated":false,"total_bytes":11363},"status":null}