{"data":{"kind":"file","path":"README.md","version_id":"ddumnq75jet6g6lyaht9q9ej","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3462,"modified_at":"2026-10-23T20:12:01.634000","content_hash":"781f218bd22a116551c56e3bbdd01d0aa2555ff9da89a9238a2b15c1d10b5143"},"entries":[],"content":"# premise-verify-eval\n\nDoes the model notice that a math word problem cannot be answered as stated?\n\nEach task is a generated word problem. It is either solvable or carries exactly one flaw:\n\n| Flaw | What changed | Label |\n|---|---|---|\n| `delete` | A premise the answer depends on was removed | underdetermined |\n| `contradict` | A count is stated twice with different values (`restate`), or a derived count is stated wrongly (`derived`, found only by computing) | inconsistent |\n| `unrealistic` | One stated number makes some count negative or fractional | unrealistic |\n| `ambiguate` | One stated number became a range or a vague word (\"a few\") | underdetermined |\n\nSymPy verifies every label. Each flawed item has a solvable parent that differs from it by one premise, and the bundled set also includes a control: drop an irrelevant premise and the answer stays the same.\n\n## Prompt modes\n\n- `permit` (default) names abstention as an option. The model ends with `\\boxed{<number>}`, or with `\\boxed{UNSOLVABLE: <reason>}` when the problem cannot be answered.\n- `force` is benchmark-style and never mentions abstention. Use it to measure how often a model answers a problem that has no answer.\n\n## Scoring\n\nThe reward `correct_decision` is 1 for the right number on a solvable item, or for `UNSOLVABLE` on a flawed one, and 0 otherwise.\n\nMetrics:\n\n| Metric | Meaning |\n|---|---|\n| `flagged`, `answered`, `no_decision` | The decision in the final `\\boxed{}` |\n| `answer_correct` | The committed number is right |\n| `net_t50`, `net_t75`, `net_t90` | Kalai-style net score at threshold t: +1 for a right decision, 0 for declining a solvable item, −t/(1−t) otherwise |\n| `completion_tokens`, `truncated` | Length, for overthinking analysis |\n\nThe task data carries `depth`, `n_ops`, `width`, `magnitude`, `premise_distance`, `parent_id`, and `template_id`. Discrimination, balanced accuracy, and depth slopes with cluster-bootstrap intervals are computed offline (see the repository's `premise_verify.metrics`).\n\n## Run\n\n```bash\nuv pip install -e .\nuv run vf-eval premise-verify-eval -m <model> -r 4 --env.taskset.prompt-mode permit\nuv run vf-eval premise-verify-eval -m <model> -r 4 --env.taskset.prompt-mode force\nuv run vf-eval premise-verify-eval -m <model> --env.taskset.flaws delete none   # subset\n```\n\n`vf-eval` uploads finished runs to the Prime Intellect platform unless you pass `--no-push`.\n\nWithout a Prime account, add `--env.agent.runtime.type subprocess`. Otherwise the harness is provisioned in a Prime sandbox, which exits when `PRIME_API_KEY` is unset. Reasoning models need `--sampling.max-tokens 16384` or more, because truncated responses never reach a decision.\n\nThe environment bundles a plain-chat harness, so the model gets no tools. A code-running harness would give it a calculator and change the task.\n\nverifiers imports `fcntl`, so run the environment on Linux, macOS, or WSL.\n\n## Configuration\n\n| Flag | Default | Meaning |\n|---|---|---|\n| `--env.taskset.prompt-mode` | `permit` | `permit` or `force` |\n| `--env.taskset.flaws` | all | Keep only these flaw types; `none` means solvable items |\n| `--env.taskset.data-path` | bundled `eval_v0` | Another JSONL(.gz) file from `premise_verify.build` |\n\n## Data\n\n`eval_v0` contains 4,939 items from 754 parents, built with seed 0: `python -m premise_verify.build --seed 0`. `eval_v0.report.json` records the build configuration, the counts, the discards, and the SHA-256 of the data.\n","encoding":"utf-8","truncated":false,"total_bytes":3462},"status":null}