{"data":{"kind":"file","path":"README.md","version_id":"yolxgsgolwogvdsb9b15iwyk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":16972,"modified_at":"2026-09-10T04:00:34.433000","content_hash":"b32eb1644ee8cd8597615920baf49c24609f7af26c960b814002c7c33925cd48"},"entries":[],"content":"# evmpatch-env\n\nAn execution-verified RL environment for **repairing real smart-contract exploits**, built\nfor Prime Intellect's [`verifiers`](https://github.com/PrimeIntellect-ai/verifiers) with an\n[OpenEnv](https://github.com/meta-pytorch/OpenEnv)-compatible adapter.\n\nEach task is a contract that was exploited on-chain. The policy is given the verified\nsource and a description of the incident, and must edit the source so that the held-out\nproof-of-concept exploit no longer succeeds, while the contract still compiles, still\ndispatches every function it dispatched before, and passes hidden functionality and\nsecurity tests. The reward is not a model judgement: the patch is compiled and the actual\nexploit is re-run against chain state frozen at the exploit block, inside a sandbox with no\nnetwork.\n\n**Version 0.3.0.** Ships as a verifiers v1 taskset (`uv run validate` green on the docker\nruntime for every task) and keeps the `load_environment` entry point. Four incidents (MCAI, NGP, GoldReserve, Bitallx; all\n2025), every task at `task_version` 2 with hidden security obligations and phased\nproof-of-concept tests. Tested on verifiers 0.3.1: 113 tests pass without Docker, 118 with\nthe `docker --network none` backend built. Version-2 grades reproduce byte-for-byte across\nthe `local` and Docker backends on one host, and the earlier task version was reproduced\nacross two hosts (macOS arm64 and Linux x86_64); see [`RECEIPT.md`](RECEIPT.md).\n\n---\n\n## Layout\n```\nevmpatch-env/\n├── evmpatch_env/\n│   ├── __init__.py          # load_environment(...) -> EvmPatchEnv; exports the v1 taskset\n│   ├── v1.py                # verifiers v1 taskset: EvmPatchTaskset / EvmPatchTask (setup, finalize, reward, validate)\n│   ├── grade_runtime.py     # grading inside a live v1 runtime (tar in, raw payload out, scored on the host)\n│   ├── sandbox.py           # container build + execution reward + canaries (structured result)\n│   ├── build_task.py        # PoC + verified source -> task dir (task.json, auto invariants)\n│   ├── openenv_adapter.py   # OpenEnv Environment/Action/Observation/State + FastAPI app\n│   └── rpc_replay.py        # fail-closed record/replay JSON-RPC proxy (offline forking)\n├── tasks/\n│   ├── ngp_2025_09/         # one of four tasks\n│   │   ├── task.json        #   metadata, split label, SHA-256 hash-locks, allow-list, task_version\n│   │   ├── project/         #   Foundry project the agent edits (src/, foundry.toml, lib/)\n│   │   ├── tests/           #   harness (hash-locked): poc.t.sol, manifest.json, hidden/\n│   │   ├── state/           #   frozen chain state: rpc_log.json (+ rpc_cache/)\n│   │   └── reference_patch.sol.diff   # known-good fix, CI self-test only (never shown to the agent)\n│   └── BUILD_LOG.md         # task build procedure, corpus table, constraints found while building\n├── worked_example/          # per-task case notes and graded control patches\n├── tests/                   # pytest: scorer probes, offline flip checks, controls, docker, v1 taskset\n├── Dockerfile + reward_entry.sh   # foundry+anvil, non-root, no network at runtime\n├── RECEIPT.md               # reproducibility receipts (grade hashes and method)\n├── .github/workflows/ci.yml # probes, offline flip check and controls for every task\n├── pyproject.toml           # verifiers-convention package (name, tags, deps, hatchling)\n├── HUB.md                   # Environments Hub listing notes and push steps\n└── OPENENV.md               # OpenEnv usage and how the adapter maps to the OpenEnv contract\n```\n\n## Quick start\n```bash\npip install -e .                       # verifiers>=0.3.1, datasets\n# Foundry must be on PATH for the reward: curl -L https://foundry.paradigm.xyz | bash && foundryup\n# The committed grades were produced with forge 1.7.1 (pinned in CI).\n\npython -c \"import evmpatch_env; e=evmpatch_env.load_environment(split='train'); print(len(e.dataset))\"\n\n# score a patch through the sandbox, offline:\npython -m evmpatch_env.sandbox tasks/ngp_2025_09 --patch <your Token.sol> --backend local\n\n# self-test any task against its own reference fix and print the canonical grade hash:\npython -m evmpatch_env.sandbox tasks/ngp_2025_09 --reference-patch --backend local --sha256\n\n# containment backend (no network inside the container at all):\ndocker build -t evmpatch-env:latest .\npython -m evmpatch_env.sandbox tasks/ngp_2025_09 --reference-patch --backend docker\n\npytest tests/                          # 125 tests; the docker-backed ones skip without the image\n```\n`load_environment(tasks_dir=None, split=\"train\", backend=\"local\", max_turns=30, task_ids=None)`\nreturns a verifiers `Environment`; use it with `vf-eval`, `prime eval run` or prime-rl like\nany Hub environment. Evaluate with a model:\n```bash\nuv run vf-eval evmpatch-env -m <model> -n 4 -a '{\"split\":\"train\",\"backend\":\"local\"}'\n```\n\n---\n\n## verifiers v1 taskset\nThe package exports `EvmPatchTaskset` (`evmpatch_env/v1.py`), so the verifiers v1 console\nscripts drive it directly:\n```bash\nuv pip install -e .                                          # verifiers>=0.3.1\ndocker build -t evmpatch-env:latest .                        # Foundry 1.7.1 + warmed solc, non-root\nuv run validate evmpatch-env --runtime.type docker --taskset.image evmpatch-env:latest -c 2\nuv run eval evmpatch-env --env.agent.harness.id bash --env.agent.runtime.type docker \\\n    --env.agent.max-turns 30 --model <model> -n 4 -r 4\n```\n`validate` runs the gold-patch and no-op checks for every task: the reference repair must\ngrade `solved` and the shipped source must grade `not_solved` (an `inconclusive` outcome\nfails both). On Prime, push the image with `prime images push` and pass the printed\nreference as `--taskset.image` with `--runtime.type prime`.\n\nWhat the agent sees: `/work/project`, a Foundry project with the verified source, its\nlibraries, `foundry.toml`, the proof-of-concept exploit test under `test/`, and\n`/work/run_poc.sh`, which compiles the project and runs that exploit offline against the\nfrozen chain state through the replay proxy. What scoring stages: at `finalize` the source\nfiles under `project/src` are read back; the reward then assembles a fresh episode workdir on\nthe host from the task's hash-locked harness (proof of concept, hidden functionality and\nsecurity suites, expected-test manifest) plus that source, re-verifies every harness hash,\nuploads it into a uuid-named directory of the runtime, runs both suites there, and parses the\nraw payload with the same scorer as the `local` and `docker` backends. The hidden suites and\nthe manifest never enter the agent's tree, and nothing in the container decides the grade.\nTaskset options: `--taskset.split`, `--taskset.task-ids`, `--taskset.image`,\n`--taskset.restrict-egress` (no execution-time network for the agent; grading is offline\neither way), `--taskset.grade-timeout`.\n\n## The task\nA task is a Foundry project snapshot with (1) a vulnerable contract's verified source and\n(2) a proof-of-concept exploit test that currently succeeds. The agent edits the source\nthrough four tools (`list_files`, `read_file`, `apply_patch`, `run_tests`) within a turn\nbudget, and the patch qualifies when:\n\n- **(a)** the exploit test fails for a reason the task declares (the exploit is blocked),\n- **(b)** the hidden functionality tests pass (legitimate use preserved),\n- **(c)** the hidden security obligations pass (the repair is complete),\n- **(d)** the project compiles and every originally dispatched selector is still dispatched,\n- **(e)** the diff touches only allow-listed source (never tests, `foundry.toml`, the harness).\n\n## Reward: three-valued, conjunctive\nComputed by `evmpatch_env/sandbox.py`. The outcome is one of:\n\n| outcome | reward | meaning |\n|---|---|---|\n| `solved` | 1.0 | (a) to (e) hold and no canary fired, on valid evidence |\n| `not_solved` | 0.0 | the run produced valid evidence and the patch does not qualify |\n| `inconclusive` | 0.0 | the run produced no valid evidence; flagged with a typed reason |\n\n`inconclusive` separates \"the exploit was blocked\" from \"the exploit test could not run\";\nthe second never earns reward. Reason codes: `compile_failed`, `proxy_start_failed`,\n`setup_failed`, `unrecorded_rpc`, `runner_crash`, `no_structured_report`, `missing_tests`,\n`unexpected_tests`, `tests_skipped`, `timeout`, `manifest_missing`, `backend_error`,\n`unrecognised_failure`; each carries a coarse `reason_family` for triage. Training code\nshould treat `inconclusive` as a missing observation, not a negative label.\n\nScoring reads `forge test --json` and the subprocess return code, never the human log (the\ntext parser survives only as a fallback that sets `structured=False` and can never yield\n`solved`). Test ids are fully qualified `<file>:<contract>:<fn(sig)>`, and the executed set\nmust exactly match the task's expected-test manifest (`tasks/<id>/tests/manifest.json`,\nitself hash-locked). `poc_blocked` requires the exploit oracle to have failed in the test\nbody for a reason listed in the task's `recognised_block_reasons`; an undeclared failure is\n`unrecognised_failure`, not a reward.\n\n`RewardResult` exposes every component (`poc_blocked`, `poc_failure_kind`, `hidden_all_pass`,\n`compiled`, `tests_match_manifest`, `diff_in_scope`, `abi_invariants`, `canaries`,\n`changed_files`, the parsed hidden run with per-test status and reason) so training can inspect why a patch\nfailed without changing the scalar. `RewardResult.canonical()` gives a host-independent view\nof the grade for cross-machine receipts.\n\nThe exploit oracle lives in the proof-of-concept test itself: `testExploit()` is phased\n(preconditions, attack, profit assertion), and only a failure in the attack or profit phase\ncounts as a block; a precondition failure is `inconclusive`. The vulnerable contract's\npatchable bytecode is `vm.etch`'d onto the live forked address, so the agent's source edit,\nnot a mocked contract, governs whether the exploit still works.\n\n## Containment\n- **No network at reward time.** The Docker backend runs the grader inside\n  `docker run --network none`. Frozen chain state is served by an in-container record/replay\n  proxy (`rpc_replay.py`) that fails closed: any JSON-RPC call not in the recording returns an\n  error, so a patch cannot reach the real chain. The `local` backend uses the same proxy and\n  `FOUNDRY_OFFLINE=true`.\n- **Harness is read-only and hash-locked.** `tests/**`, `foundry.toml` and `remappings.txt`\n  have their SHA-256 pinned in `task.json`. Every episode reassembles the project from the\n  task and applies the agent's patch only to `project/src`, so the policy cannot edit the\n  oracle. Hashes are re-verified before each run.\n- **Least privilege.** Non-root user (uid 1000), read-only state mount, wall-clock timeout,\n  a uuid-named work directory and an OS-assigned port per episode.\n\n## Canaries\nEach forces score 0 and is recorded in `RewardResult.canaries`:\n\n| canary | what it catches |\n|---|---|\n| `poc_tampered` | `poc.t.sol` hash differs from the lock: deleting, skipping or neutering the PoC |\n| `harness_tampered` | `foundry.toml`, `remappings.txt` or any hidden test hash mismatch |\n| `out_of_scope_edit` | patch changed a path outside `patchable_globs` (short-circuits before forge) |\n| `bricked` | exploit \"fails\" only because the contract was disabled (a hidden functionality test also fails) |\n| `residual_vulnerability` | exploit blocked and legitimate use intact, but a hidden security obligation (`manifest.hidden.security_tests`) still fails: an incomplete repair |\n| `empty_patch` | no source change at all |\n\nThe auto-generated `tests/hidden/invariants_auto.t.sol` checks ABI preservation two\nindependent ways, recorded separately in `RewardResult.abi_invariants`:\n\n* `bytecode_selectors` walks the patched runtime bytecode as EVM code (skipping PUSH\n  immediates) and requires every original selector to appear as a PUSH1..PUSH4 immediate,\n  i.e. in the dispatch table;\n* `dispatch_selectors` calls every original function and requires the contract not to answer\n  with the dispatcher's \"no such function\" (fail with empty returndata). `setUp` calibrates\n  this against the contract deployed on the fork before etching, so the assertion is\n  \"everything the real contract dispatches, the patch still dispatches\".\n\nTwo guards keep those from decaying into tautologies: `not_vacuous` (impossible selectors\nmust be reported absent) and `dispatch_control` (an impossible selector must get the \"no such\nfunction\" answer, or the contract has a catch-all fallback and the probe proves nothing).\n\n## Controls\nEvery task ships with graded control patches under `worked_example/<task>/controls/`, and\n`tests/test_controls_<task>.py` asserts them: the reference repair and an alternative\ncomplete repair grade `solved`; two demonstrated incomplete repairs grade `not_solved` with\nthe `residual_vulnerability` canary; a compile failure grades `inconclusive`; two runs of the\nsame patch produce identical canonical hashes. `tests/test_scorer_probes.py` holds ten\nsynthetic scorer probes (RPC failure, missing or extra tests, runner crash, name collisions,\ncompile failure, scope violation) plus regression tests for\npath-traversal patch keys, unrecorded RPC misses and the security-obligation canary; they run\nwithout Foundry and gate CI.\n\n## Splits and contamination\n- Each task carries a split label derived from its incident month\n  (`build_task.py::split_for_month`, re-checked in `load_environment`): `train` for incidents\n  dated 2025-12 or earlier, `test` for 2026-03 to 2026-08, with 2026-01 and 2026-02\n  quarantined. **All four tasks in this release are 2025 incidents and carry `split=\"train\"`;\n  the held-out split is empty until 2026 incidents are added.**\n- EVMBench's 40 contest codebases are out of scope for tasks. This is a corpus rule applied\n  when incidents are selected, not a check in `load_environment`; the four shipped tasks are\n  DeFiHackLabs incident contracts, not contest codebases.\n- Provenance: proof-of-concept tests derive from DeFiHackLabs (Apache-2.0; see `NOTICE`),\n  and contract sources are reproduced from public block-explorer verification. Only derived\n  task artifacts (frozen state, harness, task metadata) are shipped.\n\n## Scope and limits\n- The etched bytecode is a recompilation of the verified source, not the deployed bytecode;\n  sizes differ (for NGP, 9,364 bytes etched against 9,559 deployed), so a grade is a property\n  of the task's recompiled contract, and the receipts should be read that way.\n- Contracts with `immutable` variables, proxies and unverified sources are outside the\n  current build pipeline (`tasks/BUILD_LOG.md` records the screened-out cases); verified\n  single-file sources are the binding constraint on corpus growth.\n- `recognised_block_reasons` are literal: a correct repair that makes the exploit fail with\n  an undeclared message grades `inconclusive`, never `not_solved`, and should be triaged.\n- Version-2 grades have single-host cross-backend receipts; a second-host receipt is\n  pending. Foundry is pinned to 1.7.1 in CI because the report parser and the committed\n  hashes are verified against it.\n- On a v1 runtime the reward grades inside the container the agent worked in, in a fresh\n  directory re-staged from the host copy of the harness; a separate grading container is a\n  possible hardening step. The `local` and `docker` backends grade in a fresh\n  `--network none` container per episode.\n\n## Task build\n`evmpatch_env/build_task.py` documents and implements the pipeline:\n1. Assemble `project/` from verified source and libraries.\n2. Write `tests/poc.t.sol` from the DeFiHackLabs proof of concept, rewired to the etch and\n   state-replay model with a phased exploit oracle that flips PASS to FAIL on a fix.\n3. Auto-generate `tests/hidden/invariants_auto.t.sol` from the original contract's selectors\n   (ABI invariance) plus a compile gate, and write the hidden functionality and security\n   suites and `tests/manifest.json`.\n4. Warm frozen state with the network once: `rpc_replay.py record <rpc> state/rpc_log.json`\n   while running the whole suite, then replay offline until zero non-benign RPC misses.\n5. Emit `task.json`: metadata, split label, SHA-256 hash-locks of every harness file, the\n   `patchable_globs` allow-list and `task_version`.\n\nFinalize an assembled task directory: `python -m evmpatch_env.build_task finalize tasks/<id>`.\n\n## Environments Hub\n```bash\nprime env install <prime-username>/evmpatch-env\n```\nSee [`HUB.md`](HUB.md) for the listing notes. Nothing in this repository pushes on its own.\n\n## OpenEnv\nSee [`OPENENV.md`](OPENENV.md): either auto-wrap the verifiers environment with\n`openenv import .`, or serve the native `EvmPatchEnvironment` adapter directly.\n\n## Licence\nApache-2.0 (see `LICENSE` and `NOTICE`). Copyright 2026 Farseen Shaikh, SCAR.\n","encoding":"utf-8","truncated":false,"total_bytes":16972},"status":null}