{"data":{"kind":"file","path":"README.md","version_id":"hvxjtmd6w8xsnf9ujoppferl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2975,"modified_at":"2026-09-24T02:57:05.508000","content_hash":"49978d646e2f9562ed5d67c0056df2fe744aa00209dc132c31ae28bbc974d73b"},"entries":[],"content":"# stat-reasoning\n\nVerifiable statistical-reasoning tasks for RL training and evaluation. Every problem has an\nexact, programmatically-computed ground-truth answer, so scoring uses **no LLM grader** — a\nmodel either gets the number right (within a stated tolerance) or it doesn't.\n\nLLMs are shaky at *executing* applied statistics even when they recall the right formula. This\nenvironment measures that gap directly.\n\n## Task types\n\n| type | prompt | answer format |\n|------|--------|---------------|\n| `wilson_ci` | 95% Wilson score interval for k successes in n trials | `low, high` (4 dp) |\n| `two_prop` | Are two proportions different at 95% (non-overlapping Wilson intervals)? | `yes` / `no` |\n| `two_prop_z` | Pooled two-proportion z-statistic | z (4 dp) |\n| `sample_size` | Minimum n for a target margin of error at p = 0.5, z = 1.96 | integer |\n| `holm` | How many nulls are rejected by Holm–Bonferroni at α? | integer |\n\nProblems are generated deterministically from a seed, so runs are reproducible.\n`load_environment(num_examples=200, seed=7)`.\n\n## Scoring — and why there are two numbers\n\n- **`correct_answer`** (the reward, weight 1.0) — numeric answers must land within **±0.001**.\n- **`correct_loose`** (a metric, weight 0.0) — the same check at **±0.01**.\n\nReporting both separates **\"knows the right method\"** from **\"executed the arithmetic\nprecisely.\"** That distinction is not academic: in our first run, *every* failure by a smaller\nmodel was a small arithmetic slip inside a correctly-stated Wilson formula — answers off by\n0.001–0.005. Scored strictly that looks like statistical incompetence; the loose metric shows\nit isn't.\n\nThe scorer reads the model's **last line** first (the requested answer format), falling back to\nthe full completion, and compares the trailing number(s) against ground truth.\n\n## ⚠ Methodology note: give models a real token budget\n\nAn early run of this environment capped generation at **1024 tokens** and scored Claude Haiku 4.5\nat **30%**. The same model, same seed, same problems, at **4096 tokens** scored **70%**. The\noriginal number was measuring truncation, not capability — reasoning tokens consumed the budget\nbefore the final answer was emitted.\n\nIf you evaluate with this environment, allow **≥4096 tokens** and check `finish_reason`. A\nstingy budget silently converts a capability benchmark into a formatting benchmark.\n\n## Difficulty\n\nWilson intervals and yes/no significance are largely saturated for frontier models (Claude\nOpus 4.8 scored 20/20 on an early 20-item draw). `two_prop_z`, `sample_size` and `holm` were\nadded to restore headroom.\n\n## Usage\n\n```bash\nuv run vf-eval stat-reasoning\nuv run vf-eval stat-reasoning -m <model> -n 20 -r 1\n```\n\n## Extending\n\nAdd a generator returning `{\"question\", \"answer\"}` and append it to `_GENERATORS`. The scorer\ninfers the comparison from the ground-truth string (yes/no, one number, or two), so most new\ntask types need no scoring changes.\n","encoding":"utf-8","truncated":false,"total_bytes":2975},"status":null}