{"data":{"kind":"file","path":"README.md","version_id":"dugumuu8vs6jzaqv9ed1v8sj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5895,"modified_at":"2026-07-19T19:01:27.178000","content_hash":"31763be5d9bfb4417fcd07e610a91a90ca71686fbcecbaaf80b1a311a9027bf6"},"entries":[],"content":"# llm-sudoku\n\nNative Verifiers v1 multi-turn Sudoku with deterministic, reusable datasets and\nnormalized blank-cell accuracy rewards.\n\n- Environment ID: `llm-sudoku`\n- Hub ID after publishing: `hyperpotatoneo/llm-sudoku@0.2.0`\n- Harness: the built-in tool-less `null` harness\n- Reward: correct latest predictions on original gaps divided by the number of\n  original gaps\n\n## Quickstart\n\nInstall this checkout and run a small local evaluation:\n\n```bash\nuv sync\nPYTHONPATH=. uv run eval llm-sudoku \\\n  --harness.id null \\\n  --model Qwen/Qwen3-4B-Instruct-2507 \\\n  -n 3 -r 1 \\\n  --no-push\n```\n\nThe checked-in [`eval.toml`](eval.toml) reproduces the bundled hard-100 task\nselection and sampler defaults. Supply the model client on the command line:\n\n```bash\nPYTHONPATH=. uv run eval @ eval.toml \\\n  --client.base-url http://localhost:8000/v1 \\\n  --client.api-key-var EMPTY_API_KEY\n```\n\n`eval` always writes `config.toml`, `results.jsonl`, and `eval.log` under a fresh\noutput directory. The config disables platform upload; remove `--no-push` or set\n`push = true` only when an authenticated upload is intended.\n\n## Native v1 contract\n\n`LLMSudokuTaskset` owns typed, frozen `LLMSudokuTask` objects, scoring, metrics,\nand final diagnostics. `SudokuUser` owns the mutable `SudokuState` and processes\none assistant response after every model turn.\n\nEach response should end in one tag:\n\n```text\n<move>A1=5</move>\n```\n\nThe last valid tag in a response is processed. A prediction for an original gap\nis retained even when it conflicts with the current row, column, or box; the\nfeedback warns about the conflict without revealing oracle correctness. Reusing\na coordinate overwrites its value, and digit `0` clears it. Original clues are\nimmutable. Invalid responses still consume a game turn.\n\nThe game ends after all original gaps have predictions or after\n`--taskset.max-turns` assistant turns. Do not set the framework-level\n`--max-turns` to the same number: that safety limit is checked before the\ntaskset's stop hook and can classify the final game turn as truncation. Leave it\nunset, or set it strictly above the game limit.\n\n## Datasets\n\nThe bundled `data/hard_100_seed42.jsonl` contains 100 deterministic, uniquely\nsolvable hard puzzles. Dataset JSON and schema version 1 remain unchanged from\nthe v0 environment. Every row contains:\n\n- `prompt`: the actor-visible board and move format, with no oracle data;\n- `answer`: the solved grid serialized as compact JSON;\n- `info.oracle_solution`: the privileged 9x9 solution;\n- `info.puzzle`, `difficulty`, `blank_cells`, `num_blank_cells`, clue counts,\n  schema version, and generation seed.\n\nRelative dataset paths are resolved against the caller and then the installed\npackage, so the bundled path works from a checkout or wheel.\n\nGenerate another reusable dataset without starting an evaluation:\n\n```bash\nuv run llm-sudoku-generate \\\n  --output data/hard_100_seed42.jsonl \\\n  --num-examples 100 \\\n  --difficulty hard \\\n  --seed 42\n```\n\nMixed generation accepts `--easy-fraction`, `--medium-fraction`, and\n`--hard-fraction`. The weights are normalized and converted to exact integer\ncounts with largest remainders.\n\n## Taskset configuration\n\nAll environment-owned settings use the `--taskset.*` namespace.\n\n| Flag | Default | Meaning |\n| --- | --- | --- |\n| `--taskset.dataset-path` | generated | JSON/JSONL dataset to load. |\n| `--taskset.max-examples` | `100` | Generated count or loaded-file prefix; `-1` loads every saved row. |\n| `--taskset.seed` | `42` | Deterministic generation and mixed-order seed. |\n| `--taskset.difficulty` | `mixed` | `easy`, `medium`, `hard`, or `mixed`. |\n| `--taskset.easy-fraction` | `1/3` | Easy weight for mixed generation. |\n| `--taskset.medium-fraction` | `1/3` | Medium weight for mixed generation. |\n| `--taskset.hard-fraction` | `1/3` | Hard weight for mixed generation. |\n| `--taskset.max-turns` | `100` | Maximum processed assistant turns. |\n| `--taskset.system-prompt` | bundled prompt | Optional actor system-prompt override. |\n| `--taskset.include-value-function-prompt` | `false` | Attach critic-only privileged context to the typed task. |\n\n## Privileged value-function prompt\n\nThe actor prompt never contains the solution. By default,\n`task.value_function_prompt` is `None`. Opt in explicitly for a value-function\nconsumer:\n\n```bash\nPYTHONPATH=. uv run eval llm-sudoku \\\n  --harness.id null \\\n  --taskset.include-value-function-prompt \\\n  --no-push\n```\n\nWhen enabled, the task field contains exactly this complete critic system\nmessage, with the nine solved rows inserted as space-separated digits:\n\n```text\nYou are a value function estimating the expected final reward of the policy trajectory that follows. The policy cannot see the privileged information below.\n\n<privileged_information>\nSolved Sudoku grid:\n<nine solved rows>\n</privileged_information>\n```\n\nThe toggle changes only the environment-owned typed task field; it does not add\nthe privileged text to actor-visible messages.\n\n## Rewards and persisted diagnostics\n\n| Signal | Meaning |\n| --- | --- |\n| `rewards.blank_cell_accuracy` | Correct latest predictions divided by all original gaps. |\n| `metrics.coverage` | Gaps with a latest prediction divided by all original gaps. |\n| `metrics.accuracy_on_attempted_cells` | Correct predictions divided by attempted gaps; zero before any attempt. |\n\nThe transient `SudokuState` is intentionally excluded from serialized traces.\nBefore scoring, `finalize` writes the durable episode summary to\n`info.sudoku`: sorted predictions, turn count, correct/attempted/total cells,\npartial reward, terminal reason, terminal summary, and completion status.\n\n## Perlmutter\n\nReusable server launch and analysis helpers live under `cluster/`. The runners\nuse the same native `eval ... --harness.id null` path as local evaluation; see\n[`cluster/README.md`](cluster/README.md) for allocation and endpoint details.\n","encoding":"utf-8","truncated":false,"total_bytes":5895},"status":null}