{"data":{"kind":"file","path":"README.md","version_id":"wpx80lzdkel62n9lfm86q5rq","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3962,"modified_at":"2026-10-13T16:35:14.370000","content_hash":"f9676c6fb0713f8658fe7593a59eef9238bb6b8ac663bfcf88c21d3a81f8ddea"},"entries":[],"content":"# spatial-gym-env\n\nSpatial-Gym tests whether a model can act on a spatial plan, not just describe one. Each episode is a SPaRC\ngrid puzzle: the model must draw a line from the start node to the end node that obeys every rule on the board,\nmoving one cell per turn. The Spatial-Gym paper evaluates eight models in three settings (one-shot,\nstep-by-step, and step-by-step with backtracking) on 500 episodes. Giving vision models an image of the board\ninstead of text lowers their solve rate by 73%, and extended chain-of-thought keeps a 3-5x accuracy advantage\nover standard inference even in the step-by-step setting.\n\n## Settings\n\n| Setting | Config | What the model does |\n|---|---|---|\n| Step-by-step | `mode=\"step\"` (default) | One move per turn. Each turn shows the step number, the current position, the legal moves and the board with the path so far. The reply must end with `Final: <action>` (0=RIGHT, 1=UP, 2=LEFT, 3=DOWN). |\n| Step-by-step with backtracking | `mode=\"step\"`, `backtracking=true` | Same, but the model may step back onto the previous cell to undo a move. |\n| One-shot | `mode=\"one_shot\"` | The whole puzzle in one message (the SPaRC prompt). The reply ends with `####` and the full list of `(x, y)` coordinates. |\n\nOptions (`--env.taskset.<name>`):\n\n* `mode`: `step` or `one_shot`.\n* `backtracking`: `true` or `false` (step mode only).\n* `max_steps`: move limit per episode, default 100 as in the paper (step mode only).\n* `difficulty`: list of levels 1 to 5, for example `[1]` for the easiest puzzles. Default: all 500 test puzzles.\n* `max_invalid_retries`: how often to ask again when a reply has no action, default 2.\n\nTasks are ordered round-robin by difficulty level, so `-n 10` gives 2 puzzles per level.\n\n## Reward\n\nThe reward is 1.0 if the final path exactly matches one of the puzzle's stored solutions, and 0.0 otherwise. The\ncheck is the one Spatial-Gym already uses (`outcome_reward` in `Spatial_Gym.step`). In one-shot mode the answer\nis replayed move by move in the same environment. Metrics: `steps`, `reached_end`, `invalid_reply`, `retries`,\n`difficulty` (step mode), and `parsed_path`, `steps`, `difficulty` (one-shot mode).\n\n## Difference from the paper\n\nThe paper's harness sent each step as a new request: the system prompt plus the current state only. This\nenvironment keeps the whole episode in one conversation, which is how verifiers runs multi-turn episodes. The\nmodel therefore sees its earlier moves and replies. Scores are not directly comparable to the paper's\nstep-by-step numbers.\n\n## Run\n\n```bash\nuv pip install -e environments/spatial_gym\nvf-eval spatial-gym-env -m <model> --client.base-url <OpenAI-compatible URL> --client.api-key-var <ENV_VAR> \\\n  -n 20 --env.taskset.difficulty '[1]' --no-push\nvf-eval spatial-gym-env -m <model> ... --env.taskset.backtracking true\nvf-eval spatial-gym-env -m <model> ... --env.taskset.mode one_shot\n```\n\n## Citation\n\n```bibtex\n@misc{kaesberg2026spatialgym,\n  title  = {Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym},\n  author = {Kaesberg, Lars Benedikt and Yang, Tianyu and Bauer, Niklas and Ruas, Terry and Wahle, Jan Philip and Gipp, Bela},\n  year   = {2026},\n  eprint = {2604.09338},\n  archivePrefix = {arXiv}\n}\n\n@inproceedings{kaesberg-etal-2025-sparc,\n  title     = \"{SP}a{RC}: A Spatial Pathfinding Reasoning Challenge\",\n  author    = \"Kaesberg, Lars Benedikt and Wahle, Jan Philip and Ruas, Terry and Gipp, Bela\",\n  booktitle = \"Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing\",\n  month     = nov,\n  year      = \"2025\",\n  address   = \"Suzhou, China\",\n  publisher = \"Association for Computational Linguistics\",\n  url       = \"https://aclanthology.org/2025.emnlp-main.526/\",\n  doi       = \"10.18653/v1/2025.emnlp-main.526\",\n  pages     = \"10359--10390\"\n}\n```\n\nCode: https://github.com/lkaesberg/Spatial-Gym · Dataset: https://huggingface.co/datasets/lkaesberg/SPaRC\n","encoding":"utf-8","truncated":false,"total_bytes":3962},"status":null}