{"data":{"kind":"file","path":"README.md","version_id":"c600y4n62zd6d6puc68fm20j","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6923,"modified_at":"2026-07-16T15:10:42.140000","content_hash":"3d3c74809364de5ddf2e3adb9cf44e5fbdbe52c5ee97f57de23946190cadb77a"},"entries":[],"content":"# crossword-eval\n\n`crossword-eval` is a deterministic, single-turn textual evaluation for\nclue-free themed crosswords. It converts crossword geometry into either a\nfaithful monospaced grid or an explicit slot-and-crossing constraint graph, so\nvision capability is never required.\n\nThe primary reward is binary semantic validity. A response passes when it fills\nevery slot with the exact target-word multiset, uses each word once, and obeys\nall crossing constraints. It is not compared only to one reference layout:\nevery valid fill is accepted. The original 150-Pokémon puzzle has exactly 32\nvalid fills, all of which pass the verifier.\n\n## Dataset\n\nThe packaged dataset contains:\n\n- the user-supplied first-150-Pokémon puzzle, transcribed as a 50×47 text grid;\n- 15 generated and certified puzzles over Pokémon, chemical elements, and U.S.\n  states;\n- five increasing tiers per generated theme: 8, 12, 18, 26, and 36 words;\n- development tiers `t0`–`t3` and held-out tier `t4`;\n- a Pokémon supertier calibration ladder at 50, 75, 100, 125, and 150 words (`t5`–`t9`),\n  designed to bridge the gap to the supplied 150-word puzzle;\n- a finer 101/102/103/104/105/110/115/120-word bridge for empirical frontier\n  calibration;\n- a frozen nine-problem final benchmark with easy (50/75/100), medium\n  (101/105/110), and hard (115/120/125) indicative rungs;\n- 71 independently generated Pokémon calibration layouts: eight layouts at\n  30, 40, 50, 60, 75, 100, 110, and 125 words, plus seven at 150 words;\n- a replicated benchmark v1 with four fixed layouts at each empirically\n  measured GPT-5.6 Sol rung: 30, 110, and 125 words;\n- source snapshots, hashes, generation seeds, difficulty features, and a suite\n  manifest.\n\nThe canonical Pydantic contract is exported as\n`crossword_eval/data/schema/puzzle.schema.json`. It is IPUZ-inspired: mask and\nsolution grids correspond to puzzle/solution geometry, while the target\nmultiset, numbered slots, and explicit crossing graph make clue-free semantic\nverification unambiguous.\n\nCanonical target ordering comes from the packaged PokéAPI snapshot, the IUPAC\nperiodic table, and the U.S. Census Bureau state list. Generated layouts are\naccepted only after two independent solvers enumerate identical complete\nsolution sets.\n\n## Representations\n\n`ascii_grid` supplies a rectangular `#`/`.` mask and requests a filled grid in\n`<solution>` tags.\n\n`slot_graph` supplies slot IDs, lengths, coordinates, and 1-indexed crossing\nequations, then requests `ID=ANSWER` records in `<entries>` tags. This is the\nmost direct representation for language models.\n\nBoth representations support:\n\n- `closed`: no word bank, matching the original clue-free task;\n- `provided`: the exact word bank is included, isolating constraint solving\n  from theme recall.\n\n## Environment arguments\n\nSettings use the hosted Verifiers 0.2 taskset/harness config shape:\n\n| Field | Default | Meaning |\n| --- | --- | --- |\n| `config.taskset.puzzle_ids` | `['pokemon-gen1-150-original']` | Packaged puzzle IDs to evaluate |\n| `config.taskset.encodings` | `['ascii_grid']` | `ascii_grid`, `slot_graph`, or both |\n| `config.taskset.bank_modes` | `['closed']` | `closed`, `provided`, or both |\n| `config.harness.max_turns` | `1` | Single model response |\n\nExample:\n\n```bash\nprime eval run crossword-eval \\\n  -m openai/gpt-5.6 -n 3 -r 1 -t 4096 \\\n  --independent-scoring \\\n  -a '{\"config\":{\"taskset\":{\"puzzle_ids\":[\"pokemon-t0-8\",\"elements-t0-8\",\"states-t0-8\"],\"encodings\":[\"slot_graph\"],\"bank_modes\":[\"provided\"]},\"harness\":{\"max_turns\":1}}}'\n```\n\n## Metrics\n\n`semantic_valid` is the only weighted reward. Diagnostic metrics have zero\nweight: parse and format validity, dimension and mask validity, slot coverage,\nlength accuracy, target-word precision/recall, all-different compliance,\ncrossing consistency, filled-cell ratio, reference-cell accuracy, and error\ncount.\n\nReference-cell accuracy is diagnostic only because non-reference solutions may\nstill be completely valid.\n\n## Certification and current benchmark\n\nAll 100 packaged puzzles pass structural reconstruction. Every generated puzzle\nis additionally checked by the production CSP solver and an independent\nfinite-domain Z3 model; both solvers must enumerate identical complete solution\nsets before a layout is accepted. The original puzzle has 32 valid fills; each\nis accepted by the semantic verifier.\n\nThe 50/75/100/125/150-word supertiers have respectively 24, 4, 8, 4, and 4\nexact valid fills.\nTheir candidate sets, structural features, hashes, and dual-solver certificates\nare recorded in `crossword_eval/data/supertier-manifest.json`.\n\nThe bridge puzzles' five-candidate pools and independent certificates are\nrecorded in `crossword_eval/data/midtier-manifest.json`. The final nine-problem\nselection and intended rung bands are frozen in\n`crossword_eval/data/final-benchmark-manifest.json`.\n\nRepeated-rollout configs use independent scoring. This applies concurrency at\nthe individual rollout boundary so a request does not consume its 30-minute\nwall-clock timeout while waiting behind other requests in a grouped batch.\n\nThe current replicated benchmark uses four distinct layouts per rung, one\nAPI-only rollout per layout, no tools, 65,536 maximum completion tokens, a\n60-minute admitted-request timeout, and independent concurrency of two. With\nGPT-5.6 Sol at explicit `reasoning = { effort = \"high\" }`, the measured exact\npass rates are 4/4 at 30 words, 3/4 at 110 words, and 0/4 at 125 words. The 95%\nWilson intervals are wide at this sample size (51.0–100%, 30.1–95.4%, and\n0–49.0%), so the labels are useful rungs rather than claims of precise\npopulation accuracy.\n\nAll twelve final outcomes returned a normal model response; none timed out or\nexhausted the token budget. One 125-word request first ended with an incomplete\nprovider SSE stream and was retried successfully before scoring. The 125-word\nmodel failures were substantive: one false infeasibility claim and three\ncomplete but crossing-inconsistent assignments. The two platform-visible runs\nare [`qcl78wr8wf77inr3sytoso6m`](https://app.primeintellect.ai/dashboard/evaluations/qcl78wr8wf77inr3sytoso6m)\nand [`u24aas6tn5m09ynl1vbg9pl4`](https://app.primeintellect.ai/dashboard/evaluations/u24aas6tn5m09ynl1vbg9pl4).\n\nGLM-5.2 was used only as a lower-cost search proxy over 40 distinct layouts at\n30–75 words. Its observed rates were 7/8, 3/8, 5/8, 5/8, and 3/8. This\nnon-monotonic sequence is intentionally reported without a fitted word-count\ncurve: layout structure is an important difficulty variable, and GPT-5.6 Sol—\nnot GLM—determines the final rung labels.\n\nThe older broad frontier campaign remains available in the repository-level\n`reports/` directory, but it is not used to label benchmark v1 because it mixed\nsmaller token budgets with timeout-heavy hosted runs. The v1 labels above come\nonly from the fixed-setting, streamed, API-only confirmation panel.\n","encoding":"utf-8","truncated":false,"total_bytes":6923},"status":null}