{"data":{"kind":"file","path":"README.md","version_id":"neyaq9wg9vights7iuxdxfmq","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5560,"modified_at":"2026-09-05T02:37:20.447000","content_hash":"4f6f2fa538095db281b056b2fe72114fae1a25fe4a9b4ee8d6d9f8bec9e9d9ae"},"entries":[],"content":"# Redtape — verifiable abstention for public-benefits determinations\n\n**Does the agent know when a required fact is missing?** Most benefit-calculation evals score\nwhether the number is right. This one scores whether the agent notices it *cannot* produce a\nnumber, and says so.\n\nEach task is a household narrative that withholds exactly one fact. Ground truth comes from\n[PolicyEngine](https://github.com/PolicyEngine/policyengine-us), a real microsimulation\nengine used for actual policy analysis — not from another model's opinion.\n\n## Installs in seconds — no microsimulation engine required\n\nThis is worth stating before anything else, because it is the difference between an\nenvironment people try and one they mean to try.\n\n**Evaluation never touches PolicyEngine.** Answer keys are computed once, at generation time,\nand baked into the split; nothing on the scoring path imports the engine. So installing this\nenvironment does not pull a microsimulation engine, its dependency tree, or its parameter\ndatabase. `policyengine-us` is an optional `[generate]` extra, needed only if you want to\nbuild a *new* split from a different seed or tax year.\n\nThe runtime dependencies are `verifiers` and `pydantic`. That is the whole list.\n\n```bash\nprime env install jakpotvin/redtape@latest\n```\n\n```python\nfrom redtape import T1Taskset\nfrom redtape.envs.t1_eligibility import T1Config\n\ntaskset = T1Taskset(T1Config())   # 1,200 tasks ship with the package\n```\n\nNo API key is needed to load the tasks or to score an answer. The installed package is the\ntaskset, oracle and scoring; the baselines and test suite ship in the environment *source*\n(`prime env pull`, or the GitHub repo) and also need no key.\n\n## What makes it different\n\n- **Deterministic ground truth, no LLM judge.** Answer keys trace to named PolicyEngine\n  variables. Rewards are computed, not rated, so the same response scores the same way\n  every time — enforced by a golden-master determinism test in CI.\n- **Abstention is scored in both directions.** Every task withholds one fact, labelled into\n  three classes: the fact decides the outcome (abstaining is correct), the fact is missing\n  but does *not* decide it (**answering** is correct), or nothing is withheld. That middle\n  class is what stops \"always abstain\" from winning — it scores 0.131.\n- **Paired tasks.** 200 pairs differ in exactly one attribute, half of which should change\n  the answer and half of which should not, so a model cannot score well by being uniformly\n  cautious or uniformly confident.\n\n## What the metrics do\n\nThree headlines, reported separately. The weighted composite exists but is deliberately\n**not** the headline, so retuning a weight cannot move a published number.\n\n| | exact-match (n=780) | abstention (n=420) | pair-consistency (200 pairs) |\n|---|---:|---:|---:|\n| **Claude Opus 5** | **0.514** | **0.438** | **0.570** |\n| always_abstain | 0.000 | 0.131 | 0.000 |\n| never_abstain | 0.205 | 0.336 | 0.495 |\n| always_eligible | 0.036 | 0.343 | 0.500 |\n| never_eligible | 0.115 | 0.074 | 0.060 |\n| rules_only | 0.205 | 0.326 | 0.495 |\n| *ceiling agent* | *1.000* | *1.000* | *1.000* |\n\nNo trivial strategy exceeds 0.50 on any headline, and the ceiling agent proves every metric\nis actually achievable — until it was written, nothing established that the abstention metric\nwas reachable at all. All six are scripted rather than model-driven, so they cost nothing and\nneed no API key, and they reproduce the numbers above exactly.\n\nOne caveat, since the point of this README is not to overstate the engine-free claim: the\n*baselines* do read the federal poverty line from PolicyEngine, so running them needs\n`pip install -e \".[dev,generate]\"` from the source tree. The engine-free property belongs to\nthe **evaluation path** — loading tasks, prompting, parsing and scoring — which is what an\ninstalled `prime env install` gives you, and which was verified in a clean environment with\n`policyengine_us` absent.\n\n## The result\n\nClaude Opus 5, 1,200 tasks, no tools:\n\n| the withheld fact would change… | correct abstention |\n|---|---:|\n| SNAP eligibility — a **category** | **0.396** (38 / 96) |\n| a benefit amount — a **quantity** | **0.050** (9 / 180) |\n\nAn eight-fold gap between two classes that differ in one respect: whether the missing fact\nmoves a yes/no or a number.\n\nIt is not incapacity at the task — exact-match is 0.514 against a 0.205 best baseline. And it\nis not blanket caution — where the missing fact does *not* decide the outcome, so answering\nis correct, it answers 95.1% of the time. It is discriminating, on the wrong axis.\n\n## Scope, and what is not validated\n\nOne model, one state (California), one tax year, one prompt. Abstention labels come from a\nperturbation sweep, which can prove a fact is deciding but cannot prove one is not.\nMedicaid is computed but deliberately **unscored** — no external validation was obtainable\nfor it. A held-out split exists, has never been evaluated against, and stays that way.\n\n**Read [`docs/LIMITS.md`](docs/LIMITS.md) before citing any number from this environment.**\nIt is 30 sections, written as the work happened rather than retrofitted, and three of them\nretract errors found in my own published results — including a schema bug that was\npenalising exactly the behaviour the benchmark exists to reward, and which sat visible in\nevery report for two days before it was read correctly.\n\n---\n\nSource, full README, and the raw results files:\n**https://github.com/JakPot42/redtape** · Apache-2.0 · 255 tests, CI green\n","encoding":"utf-8","truncated":false,"total_bytes":5560},"status":null}