{"data":{"kind":"file","path":"README.md","version_id":"m0c2jqfjgoh4e3fnk3ohv16s","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8941,"modified_at":"2026-11-20T19:22:48.521000","content_hash":"599927ffde71134db2e529c18f3530dd538bc399b800abbeee4f6b1fd6aa9fc0"},"entries":[],"content":"# sigwall-rescue\n\n**An RL environment for on-chain incident response.** A wallet's private key has leaked and an\nattacker bot holds the same key. The agent has the leaked key, a clean sponsor wallet with a\nlittle ETH and a live EVM chain, and must move the wallet's value to a safe address before the\nbot takes it.\n\nThis is the defensive side of the problem: rescuing funds under an active adversary, with EIP-7702\n(type-4) transactions, private bundles and unbonding timers, on a real EVM rather than a mock.\nIt complements contract-auditing benchmarks such as EVMbench, which cover finding and exploiting\ncontract bugs; nothing here rewards attacking a protocol.\n\n- Framework: [verifiers](https://github.com/PrimeIntellect-ai/verifiers) v1 (`import verifiers.v1 as vf`), Prime Environments Hub compatible\n- Chain: Foundry's `anvil` with the Prague hardfork, one private instance per rollout\n- Tasks: generated from a seed; 4 difficulty tiers; disjoint `train` and `test` seed ranges\n- Reward: fraction of the wallet's USD value at the safe address, read from chain state\n- Validation: a reference responder scores 1.0 on every task; untouched tasks score 0.0\n\n## Why it is hard\n\nA careful human responder gets these wrong, and so do agents, because the obvious moves lose the funds:\n\n| Mistake | What happens on-chain |\n|---|---|\n| Send the victim ETH for gas, then transfer the tokens | The sweeper takes the ETH and uses it to take the tokens first |\n| Delegate the victim to a batch contract, then call it in a second transaction | Between the two blocks anyone can drive the open delegate; the bot drains it |\n| Send the right atomic rescue through the public mempool (L1) | The bot sees it and front-runs it in the same block |\n| Sign the authorization for `nonce + 1` while the sponsor sends the transaction | The authorization is skipped; the call to a code-less account \"succeeds\" and moves nothing |\n| Rescue only what is liquid | Stake, vesting, airdrops and NFTs stay behind and are lost |\n| Withdraw unbonded stake when it shows as unlocked | The bot withdraws it in the unlock block, one block earlier |\n\nThe reference solution sponsors **one** type-4 transaction per phase that delegates the victim to\n`RescueBatch` and executes the whole rescue inside it: unstake, release, claim, transfer and NFT moves.\nOn L1 it goes through the private relay. If stake has an unbonding period, it schedules a second bundle\nfor the exact unlock block.\n\n## Scored baselines\n\nScripted responders, each with one specific blind spot, run on 40 held-out seeds per tier\n(`python scripts/baselines.py --per-tier 40`). Values are the mean rescued fraction.\n\n| strategy | easy | medium | hard | expert |\n|---|---:|---:|---:|---:|\n| fund_then_transfer | 0.00 | 0.00 | 0.00 | 0.00 |\n| two_step_delegation | 1.00 | 0.00 | 0.00 | 0.00 |\n| atomic_public | 1.00 | 0.55 | 0.45 | 0.45 |\n| atomic_liquid_only | 1.00 | 0.52 | 0.40 | 0.34 |\n| late_unbond_withdraw | 1.00 | 1.00 | 0.93 | 0.92 |\n| reference | 1.00 | 1.00 | 1.00 | 1.00 |\n\nEach step up the tiers removes one shortcut. The reward is partial, so an agent that rescues most\nof the value is scored on how much it saved and is not treated as having failed outright.\n\n## Task anatomy\n\nEach task is one incident, generated deterministically from `(seed, tier)`:\n\n| Axis | easy | medium | hard | expert |\n|---|---|---|---|---|\n| Chain | L2 (sequencer, no mempool) | L1 or L2 | mostly L1 | mostly L1 |\n| Holdings | 1 liquid token | + 1 position | + 2 positions | + 3 positions |\n| Positions | none | stake / vesting / airdrop / NFT | adds staking with 3–8 block unbonding | same |\n| Attacker | ETH and token sweeper | + drains open delegates, + front-runs on L1 | + existing malicious 7702 delegation, + re-delegation guard, + unlock hunter | same |\n| Ticket | amounts and bot behaviour disclosed | disclosed | amounts often hidden, bot not profiled | same |\n| Window | 40 blocks | 40 | 40, or 60 with unbonding | 40 or 60 |\n\nThe prompt is an incident ticket with the addresses, holdings as the owner remembers them, the\nchain type, the address of the shared `RescueBatch` contract and the contract interfaces. See\n`render_prompt` in `sigwall_rescue/taskset.py` and the examples in `examples/`.\n\n## Tools\n\nOne MCP tool server per rollout (`chain_*`):\n\n| Tool | Effect |\n|---|---|\n| `chain_accounts()` | Victim, sponsor and safe addresses, balances, nonces, current delegation, head block, blocks left |\n| `chain_rpc(method, params)` | Read-only JSON-RPC: `eth_call`, `eth_getCode`, `eth_getLogs`, `eth_getStorageAt`, … |\n| `chain_encode_call(signature, args)` / `chain_decode(types, data)` | ABI helpers |\n| `chain_sign_authorization(account, contract, nonce)` | EIP-7702 authorization signed by `victim` or `sponsor` |\n| `chain_send_transaction(tx, private=false)` | Sign, send, mine one block; the attacker reacts afterwards |\n| `chain_send_bundle(transactions, target_block)` | L1 only: ordered, all-or-nothing, at the top of a chosen block |\n| `chain_wait_blocks(n)` | Let blocks pass; the attacker keeps acting |\n\n## Reward and metrics\n\n- `rescued` (reward, weight 1): USD value of the listed holdings at the safe address ÷ total value.\n  Tokens count up to the scenario amount, so a sponsor cannot inflate the score by sending its own funds.\n- Metrics: `stolen_fraction`, `full_rescue`, `blocks_used`, `chain_writes`, `tool_errors`, `sponsor_spent_eth`.\n- The rollout stops when everything is rescued or the incident window closes.\n\n## Robustness against reward hacking\n\n- The agent never gets an RPC endpoint. `chain_rpc` allows only an explicit list of read methods,\n  so `anvil_*`, `evm_*`, `hardhat_*`, `debug_*` and raw sends are rejected (tested).\n- Only `victim` and `sponsor` can sign. Contract creation from the agent is refused.\n- The score is computed from chain state by the environment, never from the agent's reply.\n- Bundles are atomic through EVM snapshots. A bundle that would revert is dropped whole and costs nothing.\n- Every task's solvability is checked by `vf-validate`'s gold hook. The hook runs the reference\n  responder against a fresh chain, using only agent-visible tools.\n\n## Running it\n\nRequires Python ≥ 3.11 and Foundry's `anvil` on `PATH`, tested with anvil 1.5.1 and 1.9.0. To install it: `curl -L https://foundry.paradigm.xyz | bash && foundryup`.\n\n```bash\npip install -e .                      # or: prime env install sigwall-rescue\npytest -q                             # 66 tests: solvability, failure modes, anti-cheat, splits\nvf-validate sigwall-rescue -n 8 --runtime.type subprocess        # gold = 1.0, noop = 0.0\nvf-run sigwall-rescue -n 8 -r 1 --model <model> --env.agent.harness.id null --env.agent.runtime.type subprocess\n```\n\nEnd-to-end check without an API key: `scripts/scripted_model.py` is an OpenAI-compatible server\nthat plays the reference policy through the real tool server.\n\n```bash\npython scripts/scripted_model.py --port 8765 &\nSCRIPTED_KEY=x vf-run sigwall-rescue -n 12 -r 1 --model scripted \\\n  --client.base_url http://127.0.0.1:8765/v1 --client.api_key_var SCRIPTED_KEY \\\n  --env.agent.harness.id null --env.agent.runtime.type subprocess\n# 12/12 episodes ok, mean reward 1.000\n```\n\nTaskset options (`--env.taskset.*`): `split` (`train` | `test`), `num_tasks` (default 400),\n`tiers` (subset of `easy, medium, hard, expert`) and `seed`. To run the tool server in its own\ncontainer, use the `Dockerfile` and set `taskset.task.tools.runtime = { type = \"docker\", image = \"…\" }`.\n\nCost: about 0.3–0.7 s of setup per rollout (anvil boot and contract deployment), with no network\naccess and no GPU on the environment side.\n\n## Example trajectories\n\n- `examples/reference_trajectories.json`: a full rescue on an easy L2 task, and a two-phase rescue\n  with unbonding on an expert L2 task (22 messages).\n- `examples/failure_public_mempool.json`: the right batch sent publicly on L1. Reward 0.0 and\n  93% of the value stolen, front-run in the same block.\n\n## Limits\n\n- The attacker is a scripted bot with the behaviours listed above. It is fast but not adaptive.\n- Gas markets are flat (1 gwei base fee); fee bidding wars are not modelled.\n- Token prices are fixed per task; there is no DEX or price impact.\n- Model results are not published yet; the calibration above uses scripted strategies.\n\n## Roadmap\n\n- `sigwall-review`: decide ALLOW / HOLD / DENY for signing requests (transfers, approvals, permits,\n  7702 delegations, upgrades), labelled by sigwall's policy engine.\n- Full incident response: detection, revocation of approvals, delegation hygiene and post-incident reporting in one episode.\n- Adaptive attacker profiles and fee competition.\n\nPrivate variants for training are available on request: larger held-out pools, custom attacker\nprofiles and chain-specific scenarios. Contact via GitHub [@Alinushu86](https://github.com/Alinushu86).\n\n## License\n\nApache-2.0, as part of [sigwall](https://github.com/Alinushu86/sigwall).\n","encoding":"utf-8","truncated":false,"total_bytes":8941},"status":null}