{"data":{"kind":"file","path":"README.md","version_id":"txver8fzaucsyk1qt7dxkdca","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7175,"modified_at":"2026-09-18T17:46:58.755000","content_hash":"7d340959c95b16fc6b0560c9d5561441a7b8c78698c18f15e7ce6d77cb06921e"},"entries":[],"content":"# Magic Sort Env\n\nA Magic Sort / water-sort reinforcement-learning environment for LLM agents.\n\nTake a shipped game mechanic, turn it into a reusable RL environment, and\ndocument the reward design well enough that another environment builder can\naudit it.\n\nStatus: **engine-tested, eval-run, trainer-verified (one LoRA run).**\n\n- Mechanics, generator, exact par, and reward decomposition are covered by\n  tests (`uv run pytest`).\n- The reward-design defenses are checked by a deterministic exploit pass\n  (`uv run python -m magic_sort_env.exploits`).\n- A full model-vs-environment eval has been run on the `easy` tier\n  (`gpt-5-nano`, 6 rollouts, 100% solve, reward 1.828 +/- 0.139) — see\n  [results/results.md](results/results.md). The environment runs end to end and\n  the reward decomposes as designed. **That eval also showed `easy` is\n  saturated for this model**, so it is a usable eval for weaker models but the\n  wrong tier for training this class.\n- A local Ollama band probe found the first trainable pair:\n  `qwen2.5:7b-instruct` on `micro` solved 4/8 after illegal-move feedback was\n  added. This is a band read, not a training result.\n- One hosted LoRA GRPO run has completed (2026-09-18): `Qwen/Qwen3.5-9B` on\n  `trivial`, 60 steps, $17.48. Train reward rose 0.60 -> ~1.80 and held-out\n  frozen-split reward rose 1.01 -> 1.66 (peak) with the illegal-move rate\n  falling throughout and zero legal-move hints shown. See\n  [results/results.md](results/results.md). That is one run on one tier;\n  no broader training claims are made.\n\n## Why This Exists\n\nMost public environments test single-turn instruction following or short tool\nloops. Magic Sort is different: it asks a model to plan through a multi-turn,\nhard-constraint state space, and its difficulty is unusually sharp: for\n`gpt-5-nano`, the `easy` tier with **one** empty bottle scored 1/6, and with\n**two** empty bottles scored 6/6. One bottle spans the entire range from\nnear-impossible to saturated, which makes the tier knobs a precise instrument\nfor putting a given model inside the useful 20-80% band.\n\n## Environment Protocol\n\nState is shown as compact JSON:\n\n```json\n{\"0\":[\"C\",\"C\",\"D\",\"B\"],\"1\":[\"D\",\"B\",\"D\",\"A\"],\"2\":[\"B\",\"C\",\"A\",\"B\"],\"3\":[\"A\",\"D\",\"C\",\"A\"],\"4\":[]}\n```\n\nRules:\n\n- Each bottle is a stack. Index `0` is the bottom; the last element is the top.\n- A move is `pour O D`.\n- A pour moves the top contiguous block of one color from bottle `O` to bottle\n  `D`.\n- A pour is legal when `D` is empty or its top color matches `O`'s top color,\n  and `D` has room.\n- Stuck bottles cannot be poured from.\n- Hidden cells are shown as `?`; they reveal when layers above them are poured\n  off.\n- Solved means every bottle is empty or full of one color.\n- Illegal moves name the reason. The first illegal move is free; later illegal\n  moves waste a turn. Repeating the same illegal move consecutively also shows\n  the current legal-move list.\n\n## Tiers\n\n| Tier | Colors | Empty Bottles | Layered Mechanics |\n|---|---:|---:|---|\n| `micro` | 2 | 3 | none |\n| `trivial` | 3 | 2 | none |\n| `easy` | 4 | 2 | none |\n| `medium` | 6 | 2 | 0-1 stuck and/or hidden bottle |\n| `hard` | 8 | 2 | 1-2 stuck bottles and 1-2 hidden bottles |\n\n**Difficulty is extremely sensitive to the empty-bottle count.** For\n`gpt-5-nano`, `easy` with 1 empty solved 1/6 and with 2 empties solved 6/6.\nRather than editing tiers, override the dial directly:\n\n```powershell\nuv run vf-eval magic_sort -a '{\\\"tier\\\":\\\"easy\\\",\\\"n_empty\\\":1}' --provider openai -m gpt-5-nano -n 3 -r 2 --disable-tui --disable-env-server\n```\n\n`n_colors`, `n_empty`, and `depth` all override the selected tier, so a single\ntier can be walked across the 20-80% band for whatever model you are targeting.\n\n## Reward\n\nThe reward is a weighted sum, but scoring is server-authoritative: the rubric\nreplays the model's parsed `pour O D` moves from the initial true board.\n\nSolved rollouts:\n\n- `+1.0` for solving.\n- `+1.0 * min(1, par / moves)` for efficiency.\n- `+0.2` for clean command formatting.\n\nFailed rollouts:\n\n- No solve or efficiency credit.\n- `+0.45 * progress`, where progress is the average longest same-color run per\n  bottle, normalized.\n- `+0.05` when a genuine dead end is correctly called.\n- `+0.2` for clean command formatting.\n\nWhy this shape:\n\n- Efficiency pays only on solve, so it cannot become a consolation farm.\n- Progress credit is capped below the solve payout, so failed rollouts can have\n  gradient without beating success.\n- Format is low weight. It teaches the action grammar early and then should stop\n  mattering once all rollouts format correctly.\n- Dead ends terminate immediately. The first version burned 33 extra model turns\n  after a puzzle was already bricked.\n\nFor hidden-layer puzzles, `par` is omniscient: the solver sees the true board.\nThat is intentional. The gap between omniscient par and actual moves is the\nuncertainty-handling signal.\n\n## Usage\n\nInstall locally:\n\n```powershell\ncd magic-sort-env\nuv sync --extra dev\n```\n\nRun tests:\n\n```powershell\nuv run pytest\n```\n\nRun a cheap smoke eval:\n\n```powershell\nuv run vf-eval magic_sort -a '{\\\"tier\\\":\\\"trivial\\\",\\\"num_train_examples\\\":3,\\\"num_eval_examples\\\":3}' --provider openai -m gpt-5-nano -n 1 -r 2 --disable-tui --disable-env-server --save-results\n```\n\nPowerShell requires the escaped JSON quotes shown above.\n\nRun the framework-free smoke script:\n\n```powershell\nuv run python -m magic_sort_env.core --tier easy --examples 3\n```\n\n## Results\n\nSee `results/results.md`. All results are small-sample smoke reads unless stated\notherwise. The repo intentionally reports wall-clock and token cost beside\nreward and pass rates.\n\n## Honest Limits\n\n- Exactly **one training run** backs the trainer-verified claim: LoRA GRPO,\n  one model (`Qwen3.5-9B`), one tier (`trivial`), 60 steps. The final-step\n  eval dipped below the step-45 peak (1.46 vs 1.66); no claim is made about\n  longer runs, other tiers, or full fine-tunes. See\n  [results/results.md](results/results.md).\n- Hosted-training packaging is constrained by the platform image: the wheel\n  deliberately declares no `verifiers` dependency (see `pyproject.toml`).\n  Local development installs the toolchain via `uv sync --extra dev`.\n- `results/` numbers are small-sample. Ollama-served models are quantized.\n- Hidden layers and stuck bottles are unit-tested but have not been observed in\n  a live model rollout.\n- A `reveal_count` bug (counted every pour as a reveal) was found and fixed on\n  2026-09-17; earlier reveal figures in the git history are wrong.\n- The current `vf-eval` path uses `--disable-env-server` on Windows. Saving\n  results fails for model names containing a colon on Windows (path syntax).\n- No claims are made about skill transfer from Magic Sort to other domains.\n- Hidden-layer `par` is omniscient and documented as such.\n\n## What Would Pay For The Next Version?\n\nAn environment vendor or lab looking for hard, multi-turn, partially observable\nplanning tasks with an explicit exploit catalogue. The next version worth money\nis not more water-sort polish; it is the same design discipline applied to\neconomic environments where agents manage scarce resources, credit, and\nadversaries.\n","encoding":"utf-8","truncated":false,"total_bytes":7175},"status":null}