{"data":{"kind":"file","path":"README.md","version_id":"sij1snanyhhs639viy62jis1","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3705,"modified_at":"2026-08-05T05:30:56.797000","content_hash":"474518eee962ac4f03684a4906cf04c97fa09c25a73b461cb089c4418bbc972c"},"entries":[],"content":"# pandas-agent-v1\n\nMulti-turn pandas competency agent on **verifiers.v1**. The agent gets a sandbox with pandas\n3.0.5, a *visible* sample fixture and a stub `solution.py`; it explores with `run_python` /\n`run_bash`, then `submit`s. Reward is **binary**: 1.0 iff the final submission passes every\nhidden test, run against **held-out data the agent never saw**.\n\n- **43 tasks**, 8 families: dtypes/missing values · Copy-on-Write · index alignment · groupby ·\n  merge/join · MultiIndex · time & DST · ordering and ties.\n- **Ported from** [`../pandas_agent`](../pandas_agent) (classic `verifiers`), on the steer that\n  Prime Intellect staff gave in `#environments-hub` (2026-08-03): *\"build on top of verifiers\n  v1, which is a lot more robust + you don't need to use primitives such as\n  `from prime_sandboxes import CreateSandboxRequest`.\"*\n\n## Why the tasks are hard\n\nNothing is withheld that the sandbox cannot answer in one command — but the prompt states the\n**goal**, never the pandas call. Two earlier prototypes of ours stated the algorithm and a free\n120B model scored 44/44 and 13/14. Here the visible fixture deliberately **omits the edge case**,\nso a plausible-from-memory implementation works throughout exploration and fails on the holdout.\n\nMany discriminators are pandas **3.0** behaviour changes, measured not remembered:\n\n| behaviour | pandas 3.0.5 | the common 2.x prior |\n|---|---|---|\n| text column dtype | `StringDtype` | `object` |\n| Copy-on-Write | on by default | opt-in |\n| chained assignment | raises `ChainedAssignmentError` | warns, silently no-ops |\n| `groupby` on a Categorical | `observed=True` | `observed=False` |\n| `df.values[i,j] = x` | **silently discarded, no warning** | writes through |\n| merge with a missing key on both sides | **matches** | (SQL intuition: never matches) |\n\n## The control\n\nEvery task carries three solutions: `reference` (must pass), **`naive` (must FAIL)**, and\n`stub`. A task whose plausible implementation passes has no discriminating power, and the suite\nfails the build on it — that check killed three tasks before any eval time was spent on them.\n\n```bash\npython3 ../pandas_agent/selftest.py   # 401 assertions — task content, in containers\npython3 selftest.py                   # 200 assertions — the v1 porting layer, static\npython3 smoketest.py                  #   9 assertions — real containers via DockerRuntime\npython3 smoketest.py --all            #   the full 43 (slow: a container per candidate)\n```\n\n⭐ **No Prime credits are needed to run any of this.** `verifiers.v1` ships a `DockerRuntime`\nalongside the Prime and Modal ones, so the environment develops and evaluates entirely locally.\nThe billing wall that blocked the classic version's `prime_sandboxes` path never applied here.\n\n⛔ The first suite is the real instrument. The task content is shared verbatim between the two\nenvironments, so it certifies this one too.\n\n## Containment, and an honest caveat\n\nIn the classic version the hidden tests ran in a **separate** container. Here `evaluate` runs in\nthe **same** sandbox via `runtime.run`, so containment is **temporal**: the holdout and the test\nfile are written only at scoring time, after the rollout has ended.\n\n⚠️ That is a weaker guarantee. It holds because the model cannot act after its last turn. It\nwould **not** hold against a solution that leaves a background process running in the sandbox.\n\n## Usage\n\n```bash\nprime eval run pandas-agent-v1\nprime eval run pandas-agent-v1 --taskset.families '[\"A\",\"B\"]'\nprime eval run pandas-agent-v1 --taskset.task_ids '[\"e3_missing_key_matches_missing_key\"]'\n```\n\nAn unknown family or task id **raises** rather than silently yielding an empty run.\n","encoding":"utf-8","truncated":false,"total_bytes":3705},"status":null}