{"data":{"kind":"file","path":"README.md","version_id":"rk1qebskvtay76f36scg15gl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10501,"modified_at":"2026-09-04T04:30:05.776000","content_hash":"efef2cacafe587e9ad9a55a895339f002868bf1c0a6551b680f286bffff2e9ba"},"entries":[],"content":"# break-the-grader\n\n**Break the grader, or prove it cannot be broken.**\n\nThe model is shown a quantity, a *certificate* proving that quantity lies in an\ninterval, and a grader. Its job is to find a value the grader **accepts** and the\ncertificate **refutes** — or to answer `NO_ATTACK` when no such value exists.\n\nThere is no answer key in this environment. No reference answer to match, no\njudge model, no tolerance anywhere in the scoring. A submission is wrong when a\nproof says it is outside the interval, which is a stronger statement than\ndisagreeing with a stored decimal.\n\n```bash\npip install break-the-grader\npython -m break_the_grader.cli gate          # the forgery battery, ~1s\npython -m break_the_grader.cli baseline      # the reference table, no API key\npython -m break_the_grader.cli tasks 5 --prompts\n```\n\n## What it scores, before you spend anything\n\nFour reference policies ship with the package. Each reads **only the prompt** —\nthe same string a model is shown — and each is graded by the same certificate.\nBelow they sit beside a real model run on the identical tasks. The policy half\nneeds no API key, no GPU and about four seconds:\n\n| player | kind | n | mean reward | solved | false claims | impossible | razor | narrow | wide |\n|---|---|---:|---:|---:|---:|:--|:--|:--|:--|\n| `never` — always refuse | policy | 120 | -0.350 | 32% | 81 | 39/39 | 0/21 | 0/39 | 0/21 |\n| `always` — attack blindly at the first double past the interval | policy | 120 | +0.558 | 56% | 0 | 0/39 | 14/21 | 32/39 | 21/21 |\n| `naive` — the float mistake: submit `hi + tol/2` | policy | 120 | +0.258 | 49% | 28 | 0/39 | 11/21 | 28/39 | 20/21 |\n| `careful` — do the arithmetic properly, in exact rationals | policy | 120 | +1.000 | 100% | 0 | 39/39 | 21/21 | 39/39 | 21/21 |\n| **claude-opus-5** | effort low | 120 | **+0.950** | 97% | 2 | 37/39 | 20/21 | 39/39 | 20/21 |\n| **claude-sonnet-5** | effort low | 115 | **+0.670** | 81% | 16 | 34/37 | 14/20 | 26/38 | 19/20 |\n| **claude-haiku-4-5** | no effort param | 120 | **+0.317** | 51% | 23 | 25/39 | 5/21 | 17/39 | 14/21 |\n\nRead the columns, not the mean. Blind play gets 56% and **zero of 39 impossible\nrungs** — that column is the whole of what this environment measures, and a mean\nwhich ignores it can be bought by guessing.\n\nThe model rows are a real run — 360 calls, $1.92 of a $4.00 cap reserved\nworst-case before every call — on the **same 120 seeds** as the policies. Row n\nis a function of n alone, so those are literally the same tasks, not a comparable\nsample. Replies truncated by the caller's own `max_tokens` and model refusals are\nrecorded and excluded from the rates: a harness artifact is not a model outcome.\n\nThe environment separates all three frontier models — +0.950, +0.670, +0.317 —\nand the smallest scores **below the one-line blind policy**. Not because the\ntasks are hard for a program: because the blind policy makes zero false claims\nand that model makes 23. Confidence is what costs, and it is priced in its own\ncolumn.\n\n`careful` is published on purpose. This is not a puzzle that is hard for something\nwhich checks — it is a measurement of whether the answer checks, and hiding the\nsolver would misrepresent that.\n\n## Difficulty is one number, and it has a closed form\n\nA grader that accepts everything within `tol` of a stored key, checking a quantity\ncertified to width `w`, accepts a band of provably-wrong values of size\n\n```\n    band = 2·tol − w        empty exactly when tol ≤ w/2\n```\n\nso the generator does not sample a tolerance and hope: it draws the **room** it\nwants — how many representable doubles should fit in the band — and solves\n`tol = (w + room·u)/2` for it, with `u` one ulp at that magnitude.\n\n| rung | room, in doubles | share of a batch |\n|---|---|---|\n| impossible | none fits — `NO_ATTACK` is the only right answer | 30% |\n| razor | under 16 | 23% |\n| narrow | under a million | 33% |\n| wide | more | 14% |\n\nRoom, not \"certificate widths\", because a model submits a **double**, not a real\nnumber. Half this corpus is exact integers whose certificate has width zero, and\nmeasuring in widths sent every one of them to the easiest label when a `1e-16`\ntolerance around an integer is the sharpest rung there is.\n\n**Both standing answers lose.** Always-attack fails every rung whose band is\nempty; never-attack fails every rung whose band is not. Only checking wins —\nwhich is the behaviour worth rewarding, because an auditor that always finds\nsomething is exactly as useless as one that never does.\n\n## The band is decided in exact rationals, then intersected with the doubles\n\nThose are different questions, and the gap between them is a rung.\n\nAround the integer 64 with a tolerance of `1e-15`, the band `(64, 64+1e-15)` is a\nperfectly good interval of real numbers containing **no representable double at\nall** — the nearest one is 1.4e-14 away. A model that reasons \"the tolerance is\n1e-15, so 64 + 5e-16 will do\" submits a value that *is* 64 in float64: inside the\ncertificate, not outside, and scored wrong.\n\nThat rung exists because this lab's own generator had the identical bug —\n`hi + tol/2` rounding back to `hi` on a zero-width certificate — and its battery\ncaught it before it shipped. The environment inherits the catch.\n\n## Scoring\n\n| outcome | reward | verdict |\n|---|---|---|\n| a break that verifies | **+1** | `SOLVED` |\n| `NO_ATTACK` where none exists | **+1** | `SOLVED` |\n| a value the grader accepts that is **inside** the certificate | **−1** | `WRONG` |\n| `NO_ATTACK` where a break exists | **−1** | `WRONG` |\n| a value outside the certificate the grader rejects | **0** | `UNSUPPORTED` |\n| unparseable submission | **0** | `REFUSED_PARSE` |\n\nThe two zeros are deliberate and are not the same as each other or as a wrong\nanswer. A failed attack asserts nothing false about the quantity — it is a miss.\nClaiming a break that the grader accepts and the certificate contains **is** a\nfalse claim of unsoundness, and it costs the most.\n\nThree signals come back and are kept apart: `reward` trains, `well_formed`\nseparates a refusal from a wrong answer, and `false_claim` counts the specific\nfailure this environment exists to punish. Feedback on failure is the reason and\nnothing else — no hints, no rubric, no partial credit. A witness that is nearly\nright is wrong.\n\n## The forgery battery is the test suite\n\nPlanted before any model is called: the true value submitted as an attack, both\ncertificate endpoints, a wildly wrong value, `NO_ATTACK` on a breakable grader,\nand the near-miss one ulp inside the boundary the real witness cleared. Every one\nmust fail to score.\n\n```\n686 forgeries planted · 0 leaked · GATE GREEN\n```\n\nThey live in `tests/`, beside the package rather than inside the wheel — a\ntop-level `tests/` installed into site-packages shadows everybody else's. What\nevery consumer does get is the battery **at load time**: both framework adapters\nand the CLI run it before handing back anything a model could be scored against,\nand raise if a single planted submission scores. An environment that cannot\nrefuse a forgery has no business producing a number.\n\n## The corpus\n\n104 certified quantities, each **read from a record and sha256-pinned to it** —\nChowla's cosine dip at n=4 and n=5, both Erdős #852 constants, the\nErdős–Herzog–Piranian bracket, a mode-selection threshold, and 95 conjectures\nfrom this lab's own ledger. 52 are exact integers (a tensor rank, a contact\ncount, a period count), the sharpest seeds in the set: when the true value is an\ninteger, *every* value in the tolerance window is provably wrong.\n\nEndpoints are stored as exact rationals, not decimals. An environment whose whole\nsubject is what decimals lose may not store its own facts as decimals.\n\nThe corpus grows with every certificate the lab produces, which is the part that\ncannot be copied without doing the mathematics first.\n\n## Provenance\n\nOne member of the band was not generated. The value `0.0752403861777` was\npublished for the Erdős #852 constant in that problem's own discussion thread; it\nsits `6.09e-13` outside this lab's certificate and inside any ordinary tolerance\nof it. The refutation and its certified replacement are public in that thread.\n\nThat is what the environment is about: not a hypothetical, a reproduction.\n\n## Both framework bindings are verified\n\n`break_the_grader/adapters_v0.py` exposes `load_environment` (a `SingleTurnEnv`\nwith a `Rubric`); `adapters_v1.py` exposes `BreakTheGraderTaskset` (`Task` /\n`TaskData` / `@reward`). **Both were run against a live install — on `verifiers`\n0.2.0, the version `prime` 0.6.31 pins, and on 0.3.1, the version this package's\nown Hub install command resolves — not written from a doc.** Writing them from\nthe doc had already produced three defects, two of which are silent:\n\n- a plain-string `task` column aborts every rollout in 0.2.0;\n- scoring receives pydantic message objects, not dicts, so a `.get(\"content\")`\n  misses and **every reply reads as unparseable — a whole eval reporting 0.000\n  with no error raised anywhere**;\n- `Taskset.load()` returns a list in 0.2.0 and takes an iterable in 0.3.1 — a\n  list satisfies both, which is only knowable by running both.\n\nNothing else in the package imports `verifiers`. The graders, the band geometry\nand the corpus are standard library only, and `tests/test_framework_free.py`\nproves it by blocking every third-party import and running the battery anyway.\n\n## Limits\n\nThis decides claims that reduce to finitely many exact arithmetic facts — exhibit\na value, verify an identity, bound a quantity. It does not decide mathematics at\nlarge, and a submission outside that boundary is refused rather than guessed at.\nNothing here is a formal proof in the sense of Lean or Coq. What it meets is the\nworking standard of the computer-assisted-proof tradition: one rung below a\nformal proof, and several above a decimal that looked convincing.\n\nIt is also not novel and does not claim to be. Verifier soundness is an active\n2026 literature and environments with designer-embedded reward hacks are already\non the Hub. What differs is narrower: the hacks here are not authored, they are\nminted from certificates and carry proofs, so the set is infinite, cannot be\nmemorised, and includes rungs where the honest answer is that no attack exists.\n\nMIT. Built by [cert-machine](https://carlostoledo.co) · the method and the\nmeasurement behind it: <https://carlostoledo.co/reports/gym.html>\n","encoding":"utf-8","truncated":false,"total_bytes":10501},"status":null}