{"data":{"kind":"file","path":"README.md","version_id":"p8wx1brsn0fwdebinb0az7ge","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":11042,"modified_at":"2026-09-26T06:23:19.854000","content_hash":"b866146dacde3e08280ae28206927bd89694a66c7183238b91ba7db7ce23bade"},"entries":[],"content":"# factorio-build\n\nRequires `verifiers>=0.3.1` and `factory-sim>=0.2.0`, installed from PyPI with\nprebuilt wheels.\n\nThe model writes one Python program, `def build(world):`, that builds a\nworking factory in [factory-sim](https://github.com/divagr18/factory-sim), a\ntick-exact simulator of Factorio's early game. The program runs on scenes it\nhas never seen, and the reward is the fraction of them where the factory\nverifiably produces. The program sees only what a trained RL policy sees, and\nevery call it makes is one decision of that policy's action space. The\nsimulator is checked against the real game by\n[FactorioGym](https://github.com/divagr18/FactorioGym).\n\n## Tasks\n\n`task` (`sim_task` in the v1 config) picks one of two tasks.\n\n| Task | The program has to | A scene succeeds when | Decisions |\n| --- | --- | --- | --- |\n| `construct_smelting_line` (default) | walk to an iron patch it can only partly see, choose a layout that avoids walls and water, place a burner drill on ore and a stone furnace under the drill's drop point, and fuel both | at least 10 iron plates are made by machine during a verification minute | 600 |\n| `belt_smelting` | walk between an iron patch, a coal patch and a wooden output chest, 20 to 40 tiles apart, and build a line of drills, furnaces, transport belts and burner inserters that carries plates into the chest | at least 150 iron plates reach the chest during a ten-minute window in which the program cannot act | 2,500 |\n\n`belt_smelting` starts with 4 burner drills, 4 stone furnaces, 40 belts, 10\nburner inserters and 20 coal. Twenty coal runs a line for about 79 plates, so\nthe line also needs coal from the coal patch, mined by hand or by a drill.\nPlates smelted from hand-mined ore do not count. Programs on this task hold a\nlarger `world` (the v3 profile): public markers for the three sites,\n`mine_resource`, `rotate`, `take_fuel`, `finish`, belt lanes, and each\ninserter's pickup and drop points. The prompt lists the whole API and the\nmeasured mechanics.\n\nThe task is FactorioGym's `belt_smelting` 1.1.0, ported draw for draw. Its\nreference solver delivers 150 plates in 200 of 200 episodes on the real game,\nand its port (`fsim.belt_expert`) in 599 of the first 600 simulator scenes; the\none failure has its whole coal patch on water.\n\n## Taskset\n\n- **Source:** scenes generated by `fsim.scenes.sample`, draw for draw identical\n  to FactorioGym's generator.\n- **Rows:** each row is a block of `n_scenes` consecutive scenes (default 16)\n  from one split.\n- **Splits:**\n  - `train`: seeds 0-99,999, the training layout families (four for\n    `construct_smelting_line`; `open`, `walled` and `split_patch` for\n    `belt_smelting`).\n  - `val`: seeds from 100,000 on, same families.\n  - `holdout`: families neither of the others contains (`obstructed_patch`;\n    `obstructed` and `far_chest`). Its first 100 scenes are FactorioGym's\n    frozen holdout. Evaluation only: it is built only when asked for, and never\n    as a training reward.\n- **Reward:** `success_rate`, the fraction of the row's scenes solved, plus\n  `format` at weight 0.1: +1 for a program the sandbox accepts, -1 for one it\n  refuses, 0 for no program. An optional `refusal_penalty` is off by default.\n- **Sandbox:** a static AST allowlist. No imports, no reflection, no attribute\n  starting with `_`; at most 300 lines; 2 s per scene. The rules are stated in\n  the prompt.\n- **Scoring cost:** well under a second of CPU per 16-scene row for\n  `construct_smelting_line`, and about a second for `belt_smelting`, in local\n  worker processes. No GPU, container or network.\n- **Validation:** `validate factorio-build --only-gold` checks that each row\n  is solvable: with the seed builder program for `construct_smelting_line`,\n  and with the reference builder for `belt_smelting`.\n- **Entry points:** the v1 taskset (`FactorioBuildTaskset`, Linux and macOS)\n  and the v0 `load_environment`. Both score with the same function.\n\n## Prompt versions\n\n`prompt_version` sets the last line of the user message.\n\n- `v1` (default): a plan of at most five lines, then the program. The\n  baselines below and the SFT-800 recipe used it.\n- `v2`: think the problem through step by step in plain prose, then the\n  program. The Evolve & Reinforce warm start trains on replies that reason, so\n  it asks for them.\n\n## Run it\n\nInstall from the Hub:\n\n```bash\nprime env install divagr/factorio-build\n```\n\nThe v0 entry point (`vf-eval`, `prime eval run`) takes `load_environment`'s\narguments as JSON:\n\n```bash\nprime eval run factorio-build -m openai/gpt-6-luna -n 32 -r 1 -a '{\"split\": \"val\"}'\nprime eval run factorio-build -m openai/gpt-6-luna -n 8 -r 1 \\\n    -a '{\"task\": \"belt_smelting\", \"split\": \"val\", \"prompt_version\": \"v2\"}'\nvf-eval factorio-build -m openai/gpt-6-luna -n 6 -r 1 -a '{\"split\": \"holdout\"}'   # evaluation only\n```\n\nThe v1 taskset:\n\n```bash\nuv run eval factorio-build -m openai/gpt-6-luna -n 32 -r 1 --env.taskset.split val\nuv run eval factorio-build -m openai/gpt-6-luna -n 8 -r 1 \\\n    --env.taskset.sim-task belt_smelting --env.taskset.prompt-version v2\nuv run validate factorio-build --only-gold --runtime.type subprocess --taskset.num-examples 4\n```\n\nThe v1 commands default to the `prime` runtime, which needs `prime login`.\n`--runtime.type subprocess` runs `validate` locally; in an eval\nconfig, `env.agent.runtime.type = \"subprocess\"` does the same (the recipe's\n[`eval.toml`](https://github.com/divagr18/factory-sim/tree/main/recipes/grpo-factorio-build/configs)).\n\nTraining with [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl): name\nthe taskset in the config, as the recipe's\n[configs](https://github.com/divagr18/factory-sim/tree/main/recipes/grpo-factorio-build/configs)\ndo, and set `FACTORIO_BUILD_MAX_POOLS` (for example to 7) so concurrent\nrollouts are scored in parallel instead of queueing behind one pool:\n\n```toml\n[[source]]\nname = \"factorio-build\"\nenv.taskset.id = \"factorio-build\"\nenv.taskset.split = \"train\"\nenv.taskset.sim_task = \"construct_smelting_line\"   # or \"belt_smelting\"\nenv.taskset.prompt_version = \"v2\"\nenv.taskset.task.workers = 4\n```\n\nThe full SFT + GRPO recipe, with its deploy scripts and the fixes it needed,\nis [`recipes/grpo-factorio-build`](https://github.com/divagr18/factory-sim/tree/main/recipes/grpo-factorio-build).\n\n## Results\n\nEvery number in the two tables is scored in the simulator, on\n`construct_smelting_line`.\n\n**Zero-shot baseline.** One reply per row, game notes on, 16 scenes per row,\n`prompt_version` v1, and up to 32,000 completion tokens, reasoning included.\n\n| Model | `val` scenes solved (32 rows) | `holdout` scenes solved (6 rows) | Programs accepted | Rows fully solved |\n| --- | --- | --- | --- | --- |\n| `gpt-6-luna` | 80.9% | 89.6% | 97% val, 100% holdout | 16/32 val, 2/6 holdout |\n\nThe `val` rows are subsets 0-31 at `seed=0`; the `holdout` rows are 0-5,\nfrozen indices 0-95. Six rows is a small sample. Almost every remaining\nfailure is a valid program whose plan fails on some scenes.\n\n**Trained Qwen3.5-9B.** 16 rows x 4 samples x 8 scenes per split, temperature\n0.6, thinking off. PlanBench is Blocksworld plan generation (task 1) and plan\nexecution (task 7), 500 instances each, thinking off. It measures how much\ngeneral planning a recipe costs the model.\n\n| Qwen3.5-9B | `val` (sim) | `holdout` (sim) | PlanBench generation | PlanBench execution |\n| --- | --- | --- | --- | --- |\n| base | 2.1% | 1.6% | 53.2 | 39.8 |\n| SFT-800 + GRPO (earlier recipe) | 85.9% | 84.0% | 2.8 | 3.0 |\n| Evolve & Reinforce, round 1 | 85.6% | 71.9% | 46.4 | 27.8 |\n\nBase Qwen3.5-9B solves almost nothing: half its programs exceed the sandbox's\nsize limit and most of the rest fail to parse. The earlier recipe fine-tuned\nit on 800 bare programs found by program search, then ran GRPO. It reached\n84.0% on the holdout, and PlanBench fell to 2.8 and 3.0. Evolve & Reinforce\nbuilds its warm start from the model's own explanations of those programs\n(prompt v2), mixes in replay of the base model's answers to general prompts,\nthen runs GRPO. PlanBench stays at 46.4 and 27.8, and the holdout is 12 points\nlower.\n\nModels: [`qwen3.5-9b-factorio-build-er-r1`](https://huggingface.co/divagr1925/qwen3.5-9b-factorio-build-er-r1)\n(Evolve & Reinforce, round 1) and its warm start\n[`qwen3.5-9b-factorio-build-er-r1-sft`](https://huggingface.co/divagr1925/qwen3.5-9b-factorio-build-er-r1-sft).\nAll models: [models collection](https://huggingface.co/collections/divagr1925/factoriogym-models-wip).\n\n**On the real game.** A random 16 of the SFT-800 + GRPO model's holdout\nprograms, each on 10 holdout scenes, succeed in 136 of 160 episodes on\nFactorio 2.0.60, and the simulator gives the same outcome in every episode.\nPrograms from program search succeed in 397 of 400 engine episodes (99.3%),\nagain matching the simulator in every episode. The evidence files are in\nFactorioGym.\n\n## Config\n\nv1 fields are `--env.taskset.<name>`, with dashes for underscores.\n`load_environment` takes the same names as keyword arguments, using `task` for\n`sim_task` and flattening the task-level fields.\n\n| Field | Default | Description |\n| --- | --- | --- |\n| `sim_task` | `construct_smelting_line` | `construct_smelting_line` or `belt_smelting` (`task` in `load_environment`) |\n| `split` | `train` | `train`, `val` or `holdout` (evaluation only) |\n| `n_scenes` | 16 | Scenes per row, 1 to 64 |\n| `num_examples` | 64 | Rows |\n| `seed` | 0 | First scene index of the split |\n| `game_notes` | true | Include notes on mechanics: drop point, reach, fuel, belts and inserters |\n| `prompt_version` | `v1` | `v1` (short plan, then the program) or `v2` (step-by-step prose, then the program) |\n| `task.workers` | 4 | Simulator processes per env worker; 0 runs in-process, for trusted code only |\n| `task.job_timeout_s` | 30 | Wall-clock limit per chunk of scenes; a hung worker is replaced |\n| `task.decision_budget` | the task's own | Decisions per scene: 600 for `construct_smelting_line`, 2,500 for `belt_smelting` |\n\nTo enable the refusal penalty in v1:\n`[env.taskset.task.rewards] refusal_penalty = { weight = 0.05 }`.\n\n## Changelog\n\n- 2026-09-26 (0.2.0): Adds `belt_smelting` (FactorioGym `belt_smelting`\n  1.1.0) with its v3 `world`, and `prompt_version` (`v1`, the default, or\n  `v2`). `decision_budget` now defaults to the task's own. `validate` checks\n  `belt_smelting` rows with the reference builder. Two fixes to the v0\n  `load_environment` under verifiers 0.3.1: the dataset no longer has a plain\n  string `task` column, which failed every rollout, and scene seeds are stored\n  as strings, because holdout seeds overflow Arrow's int64.\n  `construct_smelting_line` rows, v1 prompts and scores are unchanged.\n  Requires factory-sim 0.2.0.\n- 2026-09-23 (0.1.1): The sandbox accepts ordinary Python that the prompt never\n  forbade: `lambda`, `:=`, helper names with one leading underscore, and set\n  and string methods. The prompt now states the attribute and naming rules.\n  This requires factory-sim 0.1.2. Baselines are measured under these rules.\n- 2026-09-23 (0.1.0): Initial release.\n","encoding":"utf-8","truncated":false,"total_bytes":11042},"status":null}