{"data":{"kind":"file","path":"README.md","version_id":"lnwj5lbknvj8q7zizog82qus","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2524,"modified_at":"2026-09-13T04:06:06.768000","content_hash":"3eb53db4a833dd65826dd24392a3c38b4084b0a2a9e2a3ce9360bd76ae6dfdc3"},"entries":[],"content":"# nl2repobench\n\nRequires `verifiers[harbor]>=0.3.1`.\n\nGenerate an entire Python repository from a natural-language specification. Each task is graded by running the project's hidden pytest suite in a fresh per-project sandbox; reward is the resulting test pass rate.\n\n## Taskset\n\n- **Source:** Bundled per-project task directories ([`nl2repobench/test_files/`](nl2repobench/test_files)), each pairing an NL spec with a grader Docker image (`ghcr.io/multimodal-art-projection/nl2repobench/<project>`)\n- **Size:** 104 tasks\n\n## Notes\n\n- Requires a container runtime. By default, each task grades in its public Prime mirror, derived as `prime/prime/nl2repobench-<project>:1.0` from the project name. The mirrors preserve the canonical `ghcr.io/multimodal-art-projection/nl2repobench/<project>:1.0` images.\n- Grading integrity: the agent's `/workspace` is wiped at setup so the image's baked hidden tests stay out of reach, and grading strips the agent's copies of the generated test and packaging files before overlaying its source onto a fresh grading sandbox — the canonical hidden tests always come from the image.\n\nAll 104 project images have public Prime container and VM artifacts. Run a one-sample Prime smoke directly:\n\n```bash\nuv run eval nl2repobench -n 1 -r 1 \\\n  --env.agent.runtime.type prime --env.agent.runtime.vm \\\n  --env.taskset.tasks '[\"aiofiles\"]' \\\n  --no-rich --no-push\n```\n\nDocker and Modal cannot pull Prime registry references. Select the canonical GHCR images with the same template machinery:\n\n```bash\nuv run eval nl2repobench -n 1 -r 1 \\\n  --env.agent.runtime.type docker \\\n  --env.taskset.image-template 'ghcr.io/multimodal-art-projection/nl2repobench/{project}:1.0' \\\n  --env.taskset.tasks '[\"aiofiles\"]' \\\n  --no-rich --no-push\n```\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-08-30: Use deterministic public Prime mirrors for all 104 project images while keeping the image template configurable for Docker and Modal.\n- 2026-07-29: Removed the redundant hatchling `force-include` of package data dirs (already shipped via `packages`); newer hatchling errors on the duplicate archive paths.\n- 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the `Task` (a `TaskData` row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from `--env.taskset.*` to `--env.taskset.task.*`. Requires `verifiers>=0.2.0` and Python `>=3.11`.\n","encoding":"utf-8","truncated":false,"total_bytes":2524},"status":null}