{"data":{"kind":"file","path":"README.md","version_id":"uocgb0xj60eagqnsfuygxf31","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2540,"modified_at":"2026-05-18T19:23:05.255000","content_hash":"829c6b2ce6d8733fdb915785f5fa62cb127d8901d8764b348fa345674ff29c7a"},"entries":[],"content":"# Maze Subgoal Decomposition Environment\n\nTrain LLMs via GRPO to decompose grid navigation tasks into temporally ordered waypoint sequences for offline RL (STO-RL, AAMAS 2026).\n\n## Task\n\n**Input:** ASCII grid maze with start/goal positions\n**Output:** Ordered waypoints along the shortest path\n\n```\nGrid: 11x11\nStart: (0,0)\nGoal: (10,10)\nLayout:\n...........\n.####.####.\n...........\n\n→ Waypoints: [(0,0), (5,5), (10,10)]\n  Number of stages: 3\n```\n\n## Reward Function\n\nTopology-aware, fully verifiable, and aligned with the downstream STO-RL stage\npartition.  Version 0.8.9 uses turn-aware reference anchors and scores the\nstage partition induced by the proposed waypoints, not just waypoint geometry:\n\n| Component | Weight | What it checks |\n|-----------|--------|---------------|\n| Shortest-path membership | 0.08 | Waypoints lie on the start-goal shortest-path manifold |\n| Stage-map agreement | 0.08 | The induced nearest-waypoint stage map matches a progress-band reference map |\n| Reference-stage exactness | 0.06 | Cells assigned to each reference stage keep that stage label |\n| Stage-map quality | 0.10 | Stages cover the expert path, progress monotonically, and remain non-empty |\n| PBRS alignment | 0.06 | Potential-based shaping matches the reference decomposition on the expert path |\n| PBRS signal quality | 0.08 | Forward stage transitions receive stronger shaping than non-progress moves |\n| Exact stage placement | 0.20 | Waypoints match turn-aware target anchors |\n| Turn-anchor score | 0.20 | Internal waypoints land on or near turns/doorways instead of corridor midpoints |\n| Path-stage agreement | 0.08 | Stage labels along the expert path match the reference stage sequence |\n| Boundary alignment | 0.06 | Predicted stage transitions occur where the reference path stage changes |\n\nUnparseable outputs, wrong waypoint counts, invalid cells, repeated stages, and\nnon-shortest-path jumps receive 0.0.\n\n## Curriculum\n\nProcedurally generates mazes across 5 difficulty levels (5x5 to 21x21), with K=2-8 waypoints.\n\n## Usage\n\n```bash\n# Quick test\nvf-eval maze-subgoal -m Qwen/Qwen2.5-3B-Instruct -n 10 -r 1 -d -v\n\n# Full evaluation\nvf-eval maze-subgoal -m Qwen/Qwen2.5-3B-Instruct -n 100 -r 4 -s\n\n# Custom config\nvf-eval maze-subgoal -m Qwen/Qwen2.5-7B-Instruct -n 2000 -r 8 -s \\\n    -a '{\"num_samples\": 5000, \"seed\": 123}'\n```\n\n## References\n\n- STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order (AAMAS 2026)\n- PBRS: Ng, Harada & Russell (1999) — Policy invariance under reward transformations\n","encoding":"utf-8","truncated":false,"total_bytes":2540},"status":null}