{"data":{"kind":"file","path":"README.md","version_id":"e41qotbjgo8w8s2004re64dl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4023,"modified_at":"2026-10-14T07:49:54.171000","content_hash":"cac2a2fd9ef3717e530d5235509684835404ede6d503e4c96b4290e0c2bb6c5c"},"entries":[],"content":"# Coding Agent Bench v0.2.1\n\nA small executable SWE training environment for Prime Intellect Verifiers. Three synthetic, single-function Python tasks exercise issue analysis, patch editing and test writing. Each task has a buggy baseline, a public test and a separate behavioral suite. This is a demonstrator, NOT a private leaderboard or a complete coding-agent platform.\n\n## Run locally\n\nRequires Python 3.12, Docker daemon, git and the pinned `python:3.12-slim` image. The exact image digest is specified in `coding_agent_bench/core.py`. Run `docker pull python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1`, then `uv sync --python 3.12`. Smoke test: `uv run --python 3.12 python -c 'from coding_agent_bench import load_environment; print(load_environment())'`. Tests: `uv run --python 3.12 --with pytest pytest -q`.\n\n## Agent interface\n\nTools: `list_files`, `read_file`, `search_code`, `apply_patch`, `git_diff`, `run_tests`, `submit`. A patch is an exact-context `*** Begin Patch` / `*** Update File: app.py` / `@@` / `*** End Patch` edit. `*** Add File: test_agent.py` is optional for new tests. Patches never silently fuzz context; allowed files are source and a NEW agent-owned test, not public tests or config. No unrestricted shell, web, repository URL or hidden-test tool is exposed. Submit terminates the episode and exports the git diff, not a modified workspace.\n\n## Verification and scoring\n\nThe verifier creates a fresh checkout from the same Git content tree, checks the diff headers, runs `git apply`, checks file scope, safety and public API shape, then executes tests in separate Docker views. The hidden test directory is not mounted during agent-test execution. Each test produces a separate pass/fail count; all hidden and public tests must pass for `solved=true`. A new agent test earns counterfactual credit when it passes on the patched implementation but fails on the original buggy baseline. A test that passes on both earns zero test-quality credit.\n\nThe Verifiers rubric reports `verified_reward` plus zero-weight metrics for correctness, regression, test quality, patch quality, security, efficiency and solved. Partial hidden success receives bounded training credit; unsolved submissions are capped at 0.49, and integrity violations score zero. Max reward requires a valid agent-written test with counterfactual evidence. The score is a training signal, not a production software-quality metric. `patch_quality=1` is a scope whitelist, not a code-review verdict; efficiency is a simple tool-call proxy. No coverage or mutation score is implied by these fields.\n\nDocker execution uses a pinned image digest, disabled network, read-only bind mounts, read-only root, non-root UID, dropped Linux capabilities and bounded CPU/RAM/process count. Container timeouts forcibly remove timed-out containers. The Docker daemon, host filesystem, environment package source and Verifiers runtime are trusted. Static analysis only filters some obvious suspicious constructs; it is not a formal security boundary. The package embeds the behavioral suites in published Python source, so anyone can inspect them. Hidden means only inaccessible through agent tools, NOT secret from the public package. Do not use this implementation as a contamination-resistant benchmark or run it as an untrusted multi-tenant service. A private leaderboard needs an external verifier service, separately managed hidden suites and a sandbox backend available on Prime training workers; Docker availability on hosted workers has NOT been validated.\n\n## Not implemented\n\nReal-world repositories and multi-file issues, coverage / pytest-cov, mutation or property testing, persistent trajectory storage, PR review / LLM judge, NanoGPT / GPU track, external task registry, leaderboard and hosted-training validation. These need separate task and verifier infrastructure rather than claiming they exist in this MVP.\n\nPublished namespace: `pauline/coding-agent-bench`. Release version: `0.2.1`.\n","encoding":"utf-8","truncated":false,"total_bytes":4023},"status":null}