{"data":{"kind":"file","path":"README.md","version_id":"yf7z1ygrni07r5hmcit9xgll","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1799,"modified_at":"2026-07-15T14:03:36.069000","content_hash":"e23b52cd7f240072c545c835424e86b83aa6cc7c34247ad9eba066dcdaa78df4"},"entries":[],"content":"# crossword-agentic\n\nV1 Verifiers taskset for evaluating code-capable agents on answer-hidden,\nclue-free crossword constraint problems.\n\nThe taskset exposes three Pokémon puzzles with 100, 110, and 125 words. The\nagent receives a sanitized JSON specification with the word bank, slots,\ncrossings, and mask. Canonical grids are excluded. Any assignment satisfying\nthe complete word multiset, lengths, and all crossing constraints receives a\nreward of 1.\n\n## Harnesses\n\nUse a container-backed harness because tasks intentionally permit Python and\nshell execution:\n\n```bash\nuv run eval crossword-agentic --harness.id codex-native \\\n  --harness.runtime.type subprocess --model openai/gpt-5.6-sol\n\nuv run eval crossword-agentic --harness.id kimi-native \\\n  --harness.runtime.type subprocess --model moonshotai/kimi-k2.7-code\n```\n\nEach run records full multi-turn traces, tool calls, provider token usage (when\nreported), timing, exact reward, and diagnostic crossword metrics. Finished\nruns upload to the private Prime Evaluations dashboard by default.\n\n## Taskset config\n\n| Field | Default | Meaning |\n| --- | --- | --- |\n| `puzzle_ids` | 100, 110, 125 word puzzles | Ordered tasks to load |\n| `task.puzzle_path` | `puzzle.json` | Sanitized spec in the per-rollout workspace |\n| `task.answer_path` | `final_answer.txt` | Machine-checkable agent output |\n\nThe checked-in pilot configs use Verifiers' per-rollout subprocess workspaces\nbecause the available Prime Inference key is not authorized for the separate\nPrime Sandboxes API. Model calls are still team-billed by Prime Inference and\ntraces still upload to Prime Evaluations. Use the `prime` runtime when sandbox\ncredentials are available.\n\nThe public task rows and sandbox files contain no `reference_grid`, canonical\nassignment, or answer field.\n","encoding":"utf-8","truncated":false,"total_bytes":1799},"status":null}