{"data":{"kind":"file","path":"README.md","version_id":"xl05wahuy96mm648qljxpbk2","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":20769,"modified_at":"2026-09-10T23:17:51.529000","content_hash":"9b8dea457b9449efd86bbb95bbd35c11370ff9877ee9571e20ec6b29340e0f24"},"entries":[],"content":"# lattice-claims\n\n```bash\npip install lattice-claims\n```\n\n```python\nimport verifiers as vf\nenv = vf.load_environment(\"lattice-claims\")      # a SingleTurnEnv over the generator\n```\n\nStandard library only, no external tools, no data files: every instance is minted\nfrom a seed and every verdict is decided in exact rational arithmetic. **There is\nno answer key** — the environment decides submissions rather than matching them,\nso there is nothing to leak.\n\nAnd it asks for more than a verdict. A submission must declare the reference it\ndecided against, and **a right verdict reached from a reference the task did not\nstate does not score as right**. That is the whole point, and the reason the\nenvironment exists.\n\nDecide a claim about a short lattice vector exactly — or refuse it, naming the\nquantity it left out.\n\n---\n\n## The mistake this environment is built from\n\nAuditing 37 published SVP-challenge records, our grader reported **32 of them as\ndisagreeing with their published figure**. The grader was bit-exact. The finding\nwas entirely false.\n\nEvery discrepancy sat at about 1×10⁻⁴, and half a unit of norm is worth 1.7×10⁻⁴\nat those magnitudes. The published ratios had been computed from the *true* norm\nand then printed rounded; our exact ratio was computed *from the rounded norm*.\nTwo exact numbers, different references, and the comparison was meaningless.\nBoth signs of discrepancy appeared, which was the tell.\n\nExactness did not save us. **Naming what the claim was about would have.**\n\nThat is the whole thesis. A grader can be correct to the last bit and still\nwrong, because it compared against a quantity the claim was not about — so in\nthis environment a submission must declare the reference it decided against, and\na right verdict reached from the wrong reference does not score as right.\n\n> *\"A scheme with a 50-bit security level looks identical to a 128-bit scheme to\n> someone not running the estimator.\"*\n> — a parameter-selection guide, on why nobody notices\n\n---\n\n## Limits, before anything else\n\n**This certifies arithmetic about a stated lattice and a stated threshold. It\ncertifies nothing about attack cost, and it is not a claim that any deployed\nscheme is weak or strong.** Concrete security depends on cost models that the\nfield's own authors say cannot yet be pinned down precisely; nothing here\ntouches that, and a headline saying otherwise would be the one kind of\noverclaiming this project cannot afford.\n\n**The float-versus-exact story is weaker than it looks, and we measured it.** A\ncareful float grader — log domain, never converting the determinant — agrees\nwith the exact decision on every real record we have checked, 37 of 37. The\ntightest margin in the published table is 1.4×10⁻⁴ against a double's 10⁻¹⁶.\nPrecision is not where this domain breaks. Overflow is, and specification is.\n\n---\n\n## What it measures\n\nOne dial, three rungs: **how much of the reference is stated.**\n\n| rung | what the task gives | correct answers |\n|---|---|---|\n| `declared` | every quantity exact and stated | `ADMISSIBLE` / `REFUSED` |\n| `printed` | the norm as a whole number, as published tables give it | `ADMISSIBLE` / `REFUSED` / `STRADDLES` |\n| `underspecified` | a required quantity is absent | `NEEDS_DATA`, naming the field |\n\n`STRADDLES` is not a hedge. A norm printed as a whole number genuinely fails to\ndetermine the claim when the record sits within half a unit of the threshold —\nthis happens in the wild, and the environment mints it on demand.\n\nEvery task is procedural and deterministic from a seed, so nothing leaks into\npretraining and a run resumes exactly. `INFINITE = True`.\n\n### The generator's one trick\n\nFinding a short vector is the hard problem; nobody generates instances that way.\nBut a lattice can be **built around a short vector chosen first**. A\nGoldstein–Mayer lattice is fixed by its modulus `q` and entries `xs`, and a\nvector lies in it exactly when `v[0] − Σ v[i]·xs[i−1] ≡ 0 (mod q)` — one linear\ncongruence. So choose `v` freely and small, choose all but one of the `xs` at\nrandom, and solve the congruence for the last. The result is an ordinary lattice\nof determinant `q` that happens to contain a vector whose norm we picked.\n\nThe norm is *solved for*, not sampled: `v[0]` is free, so it absorbs whatever\nmakes the squared norm land on target. That is what places an instance a few\nparts in 10⁵ from the wall on demand.\n\nFloats appear in the generator and nowhere else. Choosing which instance to mint\nis not a decision about a claim, so a float may propose it; the exact predicate\nthen measures what was actually minted, and that measurement is what ships.\n\n---\n\n## Rewards\n\n| reward | weight | meaning |\n|---|---|---|\n| `certified` | scored | the verdict is the one the exact decision gives |\n| `well_formed` | 0, diagnostic | the submission declared its reference, and it is the one the task states |\n| `not_hacked` | 0, diagnostic | a float grader would have agreed |\n\nFeedback on failure is the violated relation and nothing else.\n\n---\n\n## Forgeries\n\nPlanted before any model is called. If one is accepted, the suite aborts.\n\n| forgery | must fail | why |\n|---|---|---|\n| `not_in_lattice` | `certified` | membership fails, so the claim fails |\n| `zero_vector` | `certified` | short, in the lattice, and not a solution |\n| `neighbour_lattice` | `certified` | one basis entry moved, the vector left |\n| `rounded_reference` | `well_formed` | **ours** — right verdict, decided against the rounded norm |\n| `factor_swap` | `well_formed` | decided at factor 1 while declaring 21/20 |\n| `confident_on_underspecified` | `certified` | a quantity is absent; only `NEEDS_DATA` is correct |\n| `wrong_gap_named` | `certified` | refusing is right, naming the wrong gap is not |\n| `straddle_called_definite` | `certified` | a rounded norm does not determine this claim |\n| `overflow_canary` | `certified` | `float(q)` is `inf`, so a float grader accepts anything |\n| `gap_named_in_own_schema` | `certified` | **ours, again** — `NEEDS_DATA` naming the wrong gap, with the right gap's name present only as a key of its own `reference` block |\n\n**10 planted, 10 caught, 0 accepted.**\n\n`zero_vector` earned its place by catching a hole in *our* exact grader, not in a\nmodel: `decide` alone happily accepts the zero vector, because 0 ≤ anything. The\nclaim is about a **nonzero** vector, and the predicate did not know that until a\nforgery said so.\n\n`gap_named_in_own_schema` is the second grader hole, found in review. The first\nfix for the spelling bug (below) let the grader search the *whole reply* as\nprose for the name of the absent quantity. On a task missing `claim.factor`, a\nreply saying `\"missing\": \"q\"` — the wrong gap — scored as right, because the\nword `factor` sits in its own `reference` block. The grader now reads only the\n`missing` field (wherever a model puts it, top level or inside `reference`), and\nonly the head of it — the words before any parenthesis or dash — because models\nwrite `\"q (the lattice modulus for the Gaussian heuristic ...)\"` and the\nparenthesis is explanation, not the name. The 135 stored replies were re-graded\nunder the fixed grader with no API call: **the scored reward moved on 0 rows**;\nthe diagnostic `well_formed` moved on 19 — 14 replies on the `printed` rung that\ndeclared the factor and no single norm are now well-formed, and 5 confident\nwrong answers that never named a gap no longer are.\n\n---\n\n## The canary, and what it actually shows\n\nSampled `declared` tasks, exact verdict against both float graders:\n\n| dimensions | naive float disagrees | careful float disagrees |\n|---|---|---|\n| 24 – 90 (`q` < 10³⁰⁸) | 0 / 12 | 0 / 12 |\n| 104 – 140 (`q` overflows) | **12 / 12** | 0 / 12 |\n\nIt is a **cliff, not a gradient**. A double holds ~10³⁰⁸ and a challenge-scaled\ndeterminant passes that at dimension ~102, after which `float(q)` is `inf`,\n`GH` is `inf`, and every claim is accepted. Below the cliff both float graders\nare fine.\n\nReported this way on purpose. The interesting failure in this domain is not\nrounding — it is a grader that is confidently blind, and a grader that is exact\nbut pointed at the wrong quantity.\n\n---\n\n## Layout\n\n```\nlattice_claims/\n├── taskset.py          Taskset / Task / TaskData, three rungs, grade()\n├── generate.py         procedural instances; the only file containing a float\n├── certify/exact.py    int and Fraction only; floats refused at ingest\n├── certify/naive.py    both float graders, shipped for the canary\n├── forgeries.py        planted, with the abort gate\n├── policies.py         four reference policies: the floor and the ceiling, no API key\n├── wiring.py           the second taskset, where the graph is the submission\n├── api.py              the framework-free surface every consumer shares: parse_reply,\n│                       task_row, sample, score, preflight\n├── adapters_v0.py      load_environment — the ONLY module that imports verifiers\n└── __main__.py         gate / baseline / tasks\ntests/                  27 tests across four files. Twenty run without verifiers\n                        installed; the seven binding tests SKIP rather than pass,\n                        because a binding test that passes without the framework is\n                        the same lie as a control that cannot fire.\neval/                   run_models.py (direct API), run_verifiers.py (through the\n                        framework), regrade.py, page_data.py, the stored runs\n```\n\n`import lattice_claims` is standard library only; `verifiers` is imported lazily\nand only by `adapters_v0`, which `tests/test_framework_free.py` proves by blocking\nevery third-party import and grading a submission anyway.\n\n```bash\npython3 -m pytest tests/ -q\npython3 -m lattice_claims gate               # the forgery battery\npython3 -m lattice_claims baseline --n 15    # reference policies by rung, in under a second\npython3 -m lattice_claims tasks 3 --prompts\n```\n\nThe exact predicate is a port of a decision procedure that was first used to\naudit real published records: `n = 2m` gives `(‖v‖²)^m · π^m ≤ f^n · q · m!`, and\nodd `n` clears its leftover √π against `π^(n/2)`. A certified π bracket (Machin,\nwith the truncation error accumulated rather than assumed) decides both.\n\n---\n\n---\n\n## The framework binding is verified, not written from a doc\n\n`lattice_claims/adapters_v0.py` exposes `load_environment` (a `SingleTurnEnv`\nwith a `Rubric`). **It was written against a live `verifiers` install and then run\nagainst live models — not written from documentation.** Writing one from the doc\nhad already produced three defects in a sibling environment, two of them silent:\n`load_environment` never exported from `__init__` (the Hub's own command would\nhave raised on arrival); a plain-string `task` column aborting every rollout; and\nscoring handed pydantic message objects, so a `.get(\"content\")` missed and **a\nwhole eval printed 0.000 with no error raised**. Each is designed out here and\neach has a test. `tests/test_verifiers_binding.py` SKIPS when `verifiers` is\nabsent rather than passing without it.\n\nThe rubric reports four numbers and keeps them apart on purpose: `reward` is the\nverdict, `well_formed` is whether the reference was declared and was the one the\ntask stated, `not_hacked` catches a submission that smuggles the answer in, and\n`refused_parse` separates a reply that could not be read from a wrong answer.\n\n## Run against live models\n\n`eval/run_verifiers.py` runs the real path — dataset, prompt, live model,\ncompletion, rubric — and then re-scores the framework's own completions offline.\n**72 rollouts across two models, 0 disagreements**: the framework's reward is this\npackage's reward on every one, so the framework layer owns no scoring of its own.\n`battery.py` re-scores all 72 from their stored replies on every build.\n\n| rung | Sonnet 5 · certified | · well-formed | Haiku 4.5 · certified | · well-formed |\n|---|---|---|---|---|\n| `declared` | 10/12 | 11/12 | 4/12 | **0/12** |\n| `printed` | 2/12 | 11/12 | 2/12 | 12/12 |\n| `underspecified` | 4/12 | 4/12 | 4/12 | 4/12 |\n\n36 rollouts each, effort `low` (Haiku 4.5 rejects the `effort` parameter outright,\nso it runs without it).\n\n**Read the `declared` row.** Haiku answered four of twelve correctly and declared\na usable reference on none of them. Every verdict it got right came from a\nreference it never properly stated — which is this environment's entire thesis\nappearing as a measurement rather than as a paragraph. Exactness did not save the\ngrader that started all this, and it does not save a model either: naming what you\nwere exact about is the part that carries the claim.\n\nThe `printed` rung is where both models are weakest and both are well formed —\nthey declare a reference and still miss, because the published norm is rounded and\noften does not determine the verdict at all. `STRADDLES` is the right answer there\nand it is the hardest one to say.\n\n## What this does not do\n\nIt does not propose a cryptosystem, a parameter set, or a variant of one, and it\nwill not. Auditing published arithmetic is open ground and low risk; proposing\nprimitives is crowded and high risk, and a broken proposal is unrecoverable.\nThat is a standing rule, not a judgement call made per task.\n\n## What the environment scores before anything is spent\n\nFour reference policies, on the same 45 tasks the models saw. Each reads only\nthe task's public data and is graded by the same `grade`.\n\n| policy | `declared` | `printed` | `underspecified` | overall |\n|---|---|---|---|---|\n| `exact` — decide exactly; `STRADDLES` when the half-unit window disagrees; `NEEDS_DATA` when a quantity is absent | 15/15 | 15/15 | 15/15 | **45/45** |\n| `careful` — the log-domain float grader | 15/15 | 11/15 | 0/15 | 26/45 |\n| `admissible` — always ADMISSIBLE | 10/15 | 4/15 | 0/15 | 14/45 |\n| `refused` — always REFUSED | 5/15 | 7/15 | 0/15 | 12/45 |\n\n`exact` is published on purpose: this measures whether an answer checks, not\nwhether the problem is hard for a program. `careful` is the row to read against\nthe models — a float grader that is right on every real record and has no way\nto say `STRADDLES` or `NEEDS_DATA`: its 11 is the four straddling instances, its\n0 is the cost of a grader that cannot abstain.\n\n## Results\n\n135 calls, dimensions 8–16, 15 tasks per rung, ~$1.87.\n\n**`certified` — the scored reward**\n\n| model | `declared` | `printed` | `underspecified` | overall |\n|---|---|---|---|---|\n| Opus 5 | 14/15 | **14/15** | 9/15 | **37/45** |\n| Sonnet 5 | 13/15 | 4/15 | 4/15 | 21/45 |\n| Haiku 4.5 | 7/15 | 3/15 | 8/15 | 18/45 |\n\n**`well_formed` — declared a reference, and the right one** (diagnostic)\n\n| model | |\n|---|---|\n| Opus 5 | 29/30 |\n| Sonnet 5 | 28/30 |\n| Haiku 4.5 | 19/30 |\n\n(Re-graded after the second grader fix; the first grading read 28, 21 and 13.)\n\n### What separates them\n\n**The `printed` rung, by a factor of three.** Deciding a claim from a norm that\nwas rounded before it was published — knowing when the rounding leaves the\nanswer open — is where the models come apart. On the straddling instances\nspecifically: Opus **4/4**, Sonnet **1/4**, Haiku **0/4**.\n\n**The dominant error is the same for all three, and it is the one this\nenvironment exists to train against: answering confidently when a quantity is\nabsent.** \"Said ADMISSIBLE when it was NEEDS_DATA\" is the most common failure\nfor every model — 4×, 7×, 5×. Nothing in these tasks marks them as\nunderspecified; the missing quantity has to be noticed.\n\n### The first run measured the grader, not the models\n\nWorth recording, because it is the same error the environment is named for.\n\nThe first pass scored Opus **24/45** and `well_formed` **0/30 for every model**.\nBoth were wrong. Models were correctly declaring their reference — using the\nkeys `squared_norm` and `acceptance_factor` — and the grader demanded\n`norm_squared` and `factor`, failing right answers on spelling. Worse, the\nprompt never stated the schema, so there was nothing to fail against. On the\n`underspecified` rung, 22 models had correctly returned `NEEDS_DATA` and were\nscored zero because they named the absent quantity `q` rather than the grader's\ninternal path `lattice.q`.\n\nA zero that uniform is not a result, it is a bug. Fixed on both sides: the\nprompt now states the schema, and the grader accepts any reasonable spelling of\na key or of a named quantity — a grader that insists on its own vocabulary is\ndoing precisely what this environment was built to catch.\n\n## The graph as the submission\n\nA second taskset, `lattice_claims/wiring.py`, where the model is not asked for a\nverdict but for a **wiring**: which instruments decide the claim, and what may\nreach the port that does. Building the graph *is* the grading — the two rules\nthat matter are already conditions on a wire, so nothing about correctness is\nexpressed twice:\n\n    a value that came from floating point may not enter a deciding port\n    a deciding port with nothing wired to it cannot produce a verdict\n\n45 calls, 15 tasks, dimensions 8–16.\n\n| model | attempted | wired legally | used the exact predicate | right verdict |\n|---|---|---|---|---|\n| Opus 5 | **0/15** | — | — | — |\n| Sonnet 5 | 15/15 | **15/15** | **15/15** | 9/15 |\n| Haiku 4.5 | 15/15 | **15/15** | **15/15** | 9/15 |\n\n**Composing the verifier is easy for them. Deciding is not.** Every graph either\nmodel produced was legal: no float reached a deciding port, no deciding port was\nleft dangling, and neither ever routed the tolerance grader to the sink when the\nexact predicate was available. The float screen was offered fifteen times each\nand correctly used to decide nothing.\n\nThen the verdict. **Both models routed `certified` on all fifteen** — never\n`refuted`, never `refused`. Their 9/15 is exactly the number of tasks whose\nanswer was ADMISSIBLE. They scored the base rate.\n\nThe `refused` port was on the table the whole time; it is how a graph says the\nclaim was not settled. Neither model drew that wire once, which is the same\nfailure the main taskset finds, sharpened: here abstention is a wire they chose\nnot to draw.\n\n### Opus declined this taskset\n\nAll fifteen Opus calls came back `stop_reason: refusal`, category `cyber`,\nblocked under the Usage Policy. The prompt describes lattice determinants and a\nsecurity threshold, and the classifier reads that as cyber content.\n\n**The prompt has not been reworded to get around it.** Rephrasing specifically to\nslip past a safety classifier is not something this package will do, and from the\ninside it is hard to tell that apart from rephrasing for clarity, so the wording\nstands as it is and the refusals are recorded as refusals rather than as failed\nattempts. The consequence is plain and belongs here: **as written, this taskset\ncannot evaluate Opus.**\n\n### The bug that hid the result first time\n\nThe first run scored Sonnet 1/15 and Haiku 0/15, with `gh_exact has no input\n'q*'` on 28 of 30 attempts. The catalogue printed deciding ports as `q*`, with\nthe asterisk as *our* notation for \"this port decides\" — and the models\nreasonably read the port as being named `q*`. Sonnet's very first answer was\notherwise perfect. The marker now has its own column.\n\nThat is the third time in this package that a near-zero score turned out to\nmeasure the harness. It is worth saying once more: **a zero that uniform is a\nbug, not a result.**\n\n## Reproduce\n\n```bash\npython3 -m pytest tests/ -q                       # 27 with verifiers; 20 + 7 skipped without\npython3 -m lattice_claims gate                    # the ten planted forgeries\npython3 -m lattice_claims baseline --n 15         # the reference table\npython3 eval/regrade.py [--write]                 # re-grade the stored replies, no API call\npython3 instruments/wiring/battery.py             # the whole thing, gated\n\n# spends money, never called by a battery:\npython3 eval/run_models.py --n 15 --live          # the verdict tasks, direct API\npython3 eval/run_models.py --wiring --n 5 --live  # the graph as the submission\npython3 eval/run_verifiers.py --n 36 --model claude-sonnet-5 --effort low\n```\n\n## Page\n\n`/instruments/lattice-claims`, built by `playground/lattice-claims/build.js` from\n`eval/page.json`. Every number on it is read from that record; none is typed.\n\n```bash\nPYTHONPATH=. python3 eval/page_data.py    # emits eval/page.json from the suite and the run\n(cd ../.. && node playground/build.js)\n```\n\nIt carries the two-instance diagram that explains why `STRADDLES` is a correct\nanswer — two lattices of the same dimension, both norms published as whole\nnumbers, one bar clear of the wall and one crossing it.\n","encoding":"utf-8","truncated":false,"total_bytes":20769},"status":null}