{"data":{"kind":"file","path":"README.md","version_id":"o1tylqs5dx4k4hydbbt2p86t","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":18599,"modified_at":"2026-08-13T19:54:21.461000","content_hash":"d35a9b9c9776833ae333fc4ea7ee174000134265f3329a6bc78a9aedf65ded90"},"entries":[],"content":"app-code-eval\nAn RL environment that scores application code, wired to an on-chain ownership and emission layer so that improving the agent pays the people who improve it.\n\nagentbase/app-code-eval is a Verifiers environment for the Prime Intellect Environments Hub. It runs as a normal environment (prime eval run, Hosted Training, self-hosted prime-rl), and additionally emits signed contribution receipts that an on-chain distributor on Base uses to mint each agent's token to the contributors who actually moved its verified score.\n\nagentbase is a placeholder org name. Rename the env id, package name and token ticker before the first push.\n\n\n1. Why this environment exists\nMost RL environments answer one question: is the model better? This one answers two:\n\nIs the model better? (the rubric)\nWho made it better, by how much, and on whose compute? (the receipt)\n\nQuestion 2 is what turns an environment into a launchpad. An agent minted through the protocol owns a token; the token is emitted continuously to whoever supplies verified post-training improvement; the agent's inference revenue buys that token back. The loop:\n\n        +--------------------------------------------------------+\n\n        |                                                        |\n\n        v                                                        |\n\n  token emission  ->  contributors run rollouts / supply data     |\n\n  (Bitcoin-shaped)     on Prime Intellect compute                 |\n\n        ^                          |                              |\n\n        |                          v                              |\n\n        |                 verified reward delta                   |\n\n        |                 on held-out eval split                  |\n\n        |                          |                              |\n\n        |                          v                              |\n\n   buyback pressure  <-  agent revenue  <-  agent is more useful  |\n\n   (revenue -> token)      (inference fees)                       |\n\n        |                                                        |\n\n        +--------------------------------------------------------+\n\nThe environment is the only trustable point in that loop. If the reward signal is gameable, the emission schedule funds reward hacking instead of capability. Everything below is built around that constraint.\n\n\n2. Environment card\nField\nValue\nEnv id\nagentbase/app-code-eval\nVerifiers style\nv1 taskset + agent harness (legacy load_environment also supported)\nBase class (legacy path)\nvf.StatefulToolEnv\nHarness\nbash (default), codex (optional)\nRuntime\nprime sandbox (isolated, no egress by default)\nTask unit\none application repo snapshot + one issue/spec + hidden test suite\nReward\nweighted rubric, 5 functions, scalar in [0, 1]\nMetrics (non-reward)\nturns, tool calls, tokens, wall clock, sandbox cost, diff size\nSplits\ntrain, eval_public, eval_private (rotating, never shipped in the wheel)\nLicense\nApache-2.0\n\n\n\n3. Quickstart\n# 1. install the CLI and log in\n\nuv tool install prime\n\nprime login\n\n# 2. pull the environment\n\nprime env install agentbase/app-code-eval\n\n# 3. smoke test against a hosted model\n\nprime eval run agentbase/app-code-eval \n  --model Qwen/Qwen3-8B \n  --num-examples 20 \n  --rollouts-per-example 2\n\nLocal development:\n\ngit clone https://github.com/agentbase/app-code-eval\n\ncd app-code-eval\n\nuv sync\n\nuv run vf-eval app_code_eval -n 10 -r 2\n\nprime env push          # publishes a versioned wheel to the Hub\n\nHosted Training (Verifiers v1 config):\n\n[[env]]\n\ntaskset = { id = \"agentbase/app-code-eval\", args = { split = \"train\", difficulty = \"mixed\" } }\n\nharness = { id = \"bash\", runtime = \"prime\" }\n\n[model]\n\nname = \"Qwen/Qwen3-8B\"\n\n[trainer]\n\nalgorithm = \"grpo\"\n\nLegacy (v0) config for the same environment:\n\n[[env]]\n\nid = \"agentbase/app-code-eval\"\n\nargs = { split = \"train\", difficulty = \"mixed\" }\n\nSelf-managed prime-rl uses the identical inner block under [orchestrator.train.env].\n\n\n4. The environment model\nThree parts, per the Prime Intellect environment model: dataset, harness, rubric.\n4.1 Dataset\nEach task is a frozen application repo snapshot plus a spec.\n\n{\n\n  \"task_id\": \"apc-000731\",\n\n  \"repo_url\": \"https://github.com/.../snapshot.tar.zst\",\n\n  \"commit\": \"9f2c1ab\",\n\n  \"prompt\": \"Payments webhook drops retries when the idempotency key is reused within 5s. Fix it without changing the public API.\",\n\n  \"spec\": {\n\n    \"must_pass\": [\"tests/test_webhook.py::test_idempotent_retry\"],\n\n    \"must_not_break\": [\"tests/\"],\n\n    \"public_api_frozen\": true\n\n  },\n\n  \"hidden_tests_ref\": \"sha256:...\",   // resolved inside the sandbox only\n\n  \"difficulty\": \"medium\",\n\n  \"language\": \"python\",\n\n  \"provenance\": { \"source\": \"permissive-licensed repo\", \"license\": \"MIT\" }\n\n}\n\nDesign rules:\n\nHidden tests are never shipped in the wheel. They are fetched inside the sandbox from a content-addressed store at rollout time. The model can run the public suite; it cannot read the grader.\nSnapshots are pinned by commit and hash. A task must be reproducible two years later or its historical receipts are worthless.\nProvenance is a first-class field. Only permissive licenses enter the training split. Anything ambiguous goes to eval_public at most, or is dropped.\n4.2 Harness\nApplication code cannot be scored from a single completion, so the model needs a real machine. The harness is an agent loop (bash) with runtime = \"prime\", giving each rollout a fresh Prime sandbox: container, repo checked out, network disabled except for a pinned package mirror.\n\nAvailable tools inside the sandbox:\n\nTool\nPurpose\nbash\narbitrary shell in the repo working tree\nread_file / write_file / apply_patch\nedit source\nrun_tests\nruns the public suite only, returns pass/fail plus stderr tail\nsubmit\nends the rollout and freezes the diff for grading\n\n\nStop conditions: submit called, max_turns reached (default 30), sandbox wall clock exceeded (default 15 min), or repeated identical commands (loop guard).\n\nThe legacy path uses vf.StatefulToolEnv because every rollout owns a sandbox handle:\n\nimport verifiers as vf\n\n\nclass AppCodeEnv(vf.StatefulToolEnv):\n\n    async def setup_state(self, state):\n\n        state[\"sandbox\"] = await self.pool.acquire(image=self.image)\n\n        state[\"task\"] = state[\"info\"][\"task\"]\n\n        await materialize_repo(state[\"sandbox\"], state[\"task\"])\n\n        state[\"t0\"] = time.monotonic()\n\n        return state\n\n    def update_tool_args(self, tool_name, args, messages, state):\n\n        # inject the per-rollout sandbox handle; tools stay stateless in signature\n\n        return {**args, \"sandbox_id\": state[\"sandbox\"].id}\n\n    @vf.stop\n\n    async def submitted(self, state) -> bool:\n\n        return state.get(\"submitted\", False)\n\n\ndef load_environment(split: str = \"train\", difficulty: str = \"mixed\") -> vf.Environment:\n\n    dataset = load_tasks(split=split, difficulty=difficulty)\n\n    return AppCodeEnv(\n\n        dataset=dataset,\n\n        tools=[bash, read_file, write_file, apply_patch, run_tests, submit],\n\n        rubric=build_rubric(),\n\n        max_turns=30,\n\n    )\n\nAll mutable state lives in the state dict. No globals, because rollouts run concurrently and a global would silently cross-contaminate two contributors' receipts.\n\nFor the exact Verifiers v1 taskset API, see the tasksets reference and harnesses reference. The v1 split matters here: the taskset owns tasks and rewards (the part the protocol must be able to audit), the harness owns model-facing execution (the part contributors are free to swap).\n4.3 Rubric\ndef build_rubric() -> vf.Rubric:\n\n    return vf.Rubric(\n\n        funcs=[\n\n            hidden_tests_pass,      # does it actually work\n\n            regression_guard,       # did it break anything else\n\n            spec_conformance,       # did it respect the stated constraints\n\n            static_and_security,    # ruff, mypy, semgrep, secret scan\n\n            diff_economy,           # smallest correct diff wins\n\n        ],\n\n        weights=[0.50, 0.20, 0.15, 0.10, 0.05],\n\n    )\n\nFunction\nSignal\nNotes\nhidden_tests_pass\nfraction of hidden tests passing\nthe only large-weight term; binary per test, no partial credit inside a test\nregression_guard\n0 if any previously green test turns red, else 1\nhard gate, not a gradient\nspec_conformance\nAST check on the frozen public API + JudgeRubric for prose constraints\njudge is capped at 0.15 total weight so no single LLM opinion dominates\nstatic_and_security\nruff, mypy strict on changed files, semgrep, secret scan\nsecrets or eval on user input force reward to 0\ndiff_economy\nclamp(1 - added_lines / budget)\ndiscourages rewriting the repo to satisfy tests\n\n\nLogged as metrics, not rewards (no gradient, full visibility for the emission layer): turns, tool calls, tokens in/out, sandbox seconds, USD cost, diff size, number of run_tests calls, time to first edit.\n\nThose metrics are also the anti-farming telemetry. A contributor whose rollouts show a suspicious cost/score ratio is flagged before their receipt is countersigned.\n\n\n5. Contribution receipts: from rollout to on-chain claim\nThis is the piece that turns an environment into a launchpad primitive.\n5.1 What the environment emits\nAfter each scored rollout, the environment appends a receipt:\n\n{\n\n  \"agent_id\": \"0x…\",                 // the agent whose token is being emitted\n\n  \"epoch\": 4412,\n\n  \"task_id\": \"apc-000731\",\n\n  \"split\": \"train\",\n\n  \"contributor\": \"0x…\",              // Base address, bound at run submission\n\n  \"rollout_hash\": \"sha256:…\",        // prompt + completion + tool trace\n\n  \"reward\": 0.78,\n\n  \"metrics\": { \"sandbox_seconds\": 212, \"usd\": 0.041, \"tokens\": 38104 },\n\n  \"compute\": {\n\n    \"provider\": \"prime-intellect\",\n\n    \"pod_id\": \"pod_…\",               // from the Prime compute marketplace\n\n    \"gpu\": \"H100_80GB\",\n\n    \"billing_ref\": \"…\"\n\n  },\n\n  \"attestation\": \"ed25519:…\"         // signed by the orchestrator key\n\n}\n5.2 How a receipt becomes a claim\nRollout-level reward is not what gets paid. Paying per-rollout reward pays reward hacking. What gets paid is verified improvement on a held-out split the contributor never saw.\n\nPer epoch:\n\nFreeze the candidate checkpoint produced from the epoch's accepted rollouts.\nEvaluate it on eval_private, a rotating held-out split plus canary tasks, via a hosted eval the contributors do not control.\nCompute epoch improvement ΔS = S_new - S_prev on that split. If ΔS <= 0, the epoch emits nothing. No improvement, no mint.\nAttribute ΔS across contributions with a leave-one-shard-out estimate (a cheap Shapley approximation over data shards, not over individual rollouts).\nPublish a Merkle root of (contributor, weight) to the distributor contract on Base. Contributors claim.\n\nStep 3 is the whole design. It is the difference between a training subsidy and a printing press.\n5.3 Emission schedule (Bitcoin-shaped)\nPer agent token, fixed at mint and immutable:\n\nParameter\nValue\nMax supply\n21,000,000\nEpoch length\n1 hour of protocol time\nInitial epoch subsidy\n50\nHalving\nevery 210,000 epochs (~24 years)\nPremine\n0\nEmission trigger\nΔS > 0 on the private split, otherwise the epoch subsidy is skipped, not carried forward\n\n\nSplit of each epoch subsidy (governance-tunable within hard bounds set at mint):\n\nRecipient\nShare\nRationale\nTraining contributors\n70%\nthe work being bought\nTask and environment authors\n12%\nwithout new tasks the score saturates and the flywheel stalls\nCompute providers\n10%\nsee section 6\nAgent creator\n8%\nfounder allocation, vesting, streamed per epoch, never upfront\n\n\nNote the deliberate deviation from Bitcoin: Bitcoin pays for hashes that are hard, this pays for improvement that is verified. Difficulty adjustment has an analogue here, which is the private eval getting harder as the agent saturates it. Task authors are paid precisely to keep that treadmill moving.\n5.4 Buyback\nAgent revenue (inference fees, in USDC on Base) is split by an immutable router:\n\nSink\nShare\nCompute and inference cost reserve\n50%\nBuyback of the agent's own token\n35%\nProtocol fee\n10%\nInsurance / slashing backstop\n5%\n\n\nBought-back tokens go to a locked treasury that funds future epochs, not to a burn. Burning is louder; recycling into emissions actually keeps paying trainers when the halving bites.\n\nHonest note on the flywheel: the loop only closes if the agent has real paying demand. Emission bootstraps supply of training work; it cannot bootstrap demand. An agent that never earns revenue is a token with a schedule and no buyer, and the protocol should say so on the agent's page rather than hide it. Every agent page should surface: revenue last 30d, buyback executed, ΔS per epoch, epochs skipped for zero improvement.\n\n\n6. Compute as the intermediation layer\nThe Prime Intellect compute marketplace is the settlement rail between \"someone wants to train this agent\" and \"GPUs actually ran\".\n\nTwo participation modes:\n\nA. Bring your own compute. A contributor provisions their own pod, runs rollouts against the environment, submits receipts. Payment is in the agent's token via the epoch Merkle claim.\n\nprime availability list --gpu-type H100_80GB\n\nprime pods create --gpu-type H100_80GB --gpu-count 1 --image agentbase/app-code-eval:latest\n\nprime pods status <pod_id>\n\nB. Delegated compute. A holder who does not want to operate hardware stakes tokens into an agent's training vault. The vault treasury (funded by buyback + staked capital) provisions compute through the platform API and runs the epoch on the staker's behalf. The staker earns a share of that epoch's contributor emission minus a performance fee.\n\nWhy route through the marketplace rather than build a compute layer:\n\nBilling is the oracle. Pod ids, GPU type, runtime and billing rows are queryable through the platform API, so the on-chain claim references a compute fact that can be checked, not a self-reported number.\nSandboxes are already isolated and priced per second, which is exactly the granularity the receipt needs.\nHosted Training and hosted evals give a neutral referee. The private-split evaluation in section 5.2 runs as a hosted evaluation, so no contributor grades their own homework.\n\nRelevant surfaces: GET /availability, POST /pods, GET /pods/{id}, wallet and billing history endpoints, and hosted evaluations for step 3 of the epoch.\n\nTrust boundary, stated plainly: the compute layer is a centralized platform. Receipts reference its billing data, so the protocol inherits a trusted oracle at that point. Mitigation is redundancy (a second attestation from an independent verifier re-running a sample of rollouts) plus a challenge window before any epoch root is finalized. Do not describe this as trustless. It is verifiable, which is a different and more defensible claim.\n\n\n7. Threat model\nThe environment is the mint. Anything that fools the rubric prints tokens.\n\nAttack\nMitigation\nOverfitting to visible tests\nhidden test suite resolved in-sandbox, never in the wheel\nMemorizing the private split\neval_private rotates every epoch and is never returned to the model or the contributor\nDeleting or weakening tests to pass\nregression_guard hard gate + diff inspection: test-file edits outside the allowed path force reward to 0\nSandbox escape / network exfiltration to fetch the grader\nno egress, pinned mirror, egress attempt voids the rollout and slashes the bond\nSybil contributors farming small deltas\nbonded stake per contributor address, per-address cap on epoch share, leave-one-shard-out attribution kills duplicate shards\nRollout replay\nrollout_hash dedupe across all history, receipts are single-use\nJudge collusion (contributor and judge model aligned)\njudge capped at 0.15 weight, judge model rotated, judge disagreement with hidden tests logged and audited\nTask-author self-dealing (authoring easy tasks then solving them)\ntask authorship disclosed on-chain, authors excluded from contributor emission on their own tasks\nEpoch griefing (poisoning shards so ΔS <= 0)\nshards are scored before inclusion, negative-marginal shards are rejected and the bond is slashed\n\n\nEvery item here is a reason the reward function is deliberately boring: hard gates, small judge weight, and heavy weight on tests that the model cannot read.\n\n\n8. Repository layout\napp-code-eval/\n\n├── app_code_eval/\n\n│   ├── __init__.py\n\n│   ├── environment.py       # load_environment, AppCodeEnv\n\n│   ├── taskset.py           # v1 taskset: tasks, rewards, metrics\n\n│   ├── rubric.py            # the five scoring functions\n\n│   ├── tools.py             # bash, apply_patch, run_tests, submit\n\n│   ├── sandbox.py           # Prime sandbox pool, repo materialization\n\n│   └── receipts.py          # receipt construction, signing, export\n\n├── tasks/\n\n│   ├── train.jsonl\n\n│   ├── eval_public.jsonl\n\n│   └── canaries.jsonl       # never used for training, drift detection only\n\n├── contracts/               # Base: token, distributor, revenue router, vault\n\n├── configs/\n\n│   ├── eval.toml\n\n│   └── train.toml\n\n├── tests/\n\n├── pyproject.toml\n\n└── README.md\n\n\n9. Roadmap\n- [ ] *v0.1* environment only: dataset, sandbox harness, rubric, published to the Environments Hub, no chain. Prove the reward signal is not gameable before anything is minted.\n- [ ] *v0.2* receipts and epoch attribution off-chain, published as signed JSONL. Run one agent end to end with fake tokens.\n- [ ] *v0.3* Base contracts on testnet: token, epoch distributor with Merkle claims, revenue router.\n- [ ] *v0.4* delegated compute vault, challenge window, second-verifier attestation.\n- [ ] *v1.0* launchpad: any user mints an agent, an environment, and a token in one flow.\n\nShip order matters. A launchpad on top of a gameable rubric is a faucet.\n\n\n10. Legal and compliance\nA token that grants ownership of a revenue-producing agent, is emitted on a fixed schedule, and is bought back with that revenue has a serious chance of being treated as a security in the US, the EU (MiCA) and elsewhere, whatever the whitepaper says. Buyback and revenue-share features are precisely the ones regulators look at. Get securities counsel in each target jurisdiction before launch, decide on geofencing and KYC for the launchpad, and do not treat this README as legal advice. I am not a lawyer, and nothing here is legal or investment advice.\n\n\n11. Citation\nbibtex\n@software{app_code_eval,\n\n  title  = {app-code-eval: a verified-improvement RL environment for agent ownership markets},\n\n  author = {Azzouzi, Bilal},\n\n  year   = {2026},\n\n  url    = {https://github.com/agentbase/app-code-eval}\n\n}\n","encoding":"utf-8","truncated":false,"total_bytes":18599},"status":null}