{"data":{"kind":"file","path":"README.md","version_id":"o2ftpoipc473aqq4oypkhpu2","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9541,"modified_at":"2026-09-12T11:16:28.740000","content_hash":"69b03af58d9051dcfbc109de8f37646f6ebb92ef143af3513538decda6cbbb14"},"entries":[],"content":"# pandas-agentic-v1\n\nClean, reconcile and merge-audit messy order exports inside a sandbox. **The artifact is\nthe answer** — the reward reads the file the agent wrote, not what it said — and the\ngrader never imports pandas, so pandas' own conventions are what is being tested.\n\n| | |\n|---|---|\n| **Archetype** | Agentic, sandboxed, artifact-verified, procedural + `INFINITE` |\n| **Harness** | `bash`, `claude-code`, `codex` — anything with a shell |\n| **Runtime** | Container required (`NEEDS_CONTAINER = True`); every task declares the image `amancevice/pandas:slim-3.0.5` |\n| **Reward** | `artifact_correct` (1.0), binary per task; `produced_output` metric separates \"no run\" from \"wrong run\" |\n\n## Task families\n\nEvery task starts from a generated order export with the columns `order_id, category,\nregion, units, unit_price`. Missing values appear as any of `-`, `?`, `N/A`, `NA`, `null`\nor an empty cell (pandas' default `na_values` does not cover all of these), numbers may\ncarry thousands separators or a `$`, and some rows are exact duplicates. The prompt states\nall of this; nothing about the input is a hidden trick.\n\n| Family | The work | Output | Scores 0 if |\n|---|---|---|---|\n| `repair` | messy CSV → clean typed CSV: drop exact duplicates keeping the first, empty every missing cell, `units` as integers, `unit_price` as floats rounded half-up to 2 decimals, same header and row order | `clean.csv` | any sentinel left in place, wrong value, kept duplicate, or reordered row |\n| `reconcile` | clean, then per-`category` count / **nulls** / sum / mean of `unit_price`, rounded half-up at the end; a category whose prices are all missing is reported with count 0, sum 0.0 and mean null | `result.json` | nulls averaged in, banker's rounding, a missing or phantom category |\n| `merge` | clean orders (`left.csv`) and shipments (`right.csv`), then audit a LEFT join on `order_id`: joined row count plus the unmatched keys on both sides | `result.json` | wrong row count — an order with two shipment rows contributes two joined rows — or a wrong key list |\n\nTask `idx` selects the family (`families[idx % len(families)]`). The first 60 tasks of the\ndefault seed are the fixed benchmark slice, 20 per family; the stream continues\nindefinitely for training.\n\n## Rewards and metrics\n\n`finalize` reads the output file from the sandbox while it is alive (`clean.csv` as text,\n`result.json` parsed) and records why when it cannot; the reward runs from what it\ncaptured. A missing or unparseable file scores 0.\n\n| Reward | Weight | Pays 1.0 when |\n|---|---|---|\n| `artifact_correct` | 1.0 | **repair**: the same number of rows as the reference, in the same order; per cell, a missing value must be an empty string, integers and strings must match exactly, floats must be within `task.float_tolerance` (default 0.001). **reconcile**: the set of categories equals the reference exactly; per category `count` and `nulls` are exact, `sum` and `mean` within the tolerance, and a null reference mean requires a null. **merge**: `joined_rows` exact and both key lists equal the reference as sets of strings (their order is not enforced). Anything else scores 0. |\n\nThe tolerance is deliberately below 0.005. A banker's-rounded value sits 0.01 from the\nhalf-up one and an unrounded x.xx5 value 0.005 from it; both must fail regardless of\nbinary float noise, which a tolerance at exactly 0.01 or 0.005 would not guarantee.\n\n| Metric | Meaning |\n|---|---|\n| `produced_output` | 1.0 if an output file was read at all (parseable JSON, or any CSV text) — separates \"never wrote the file\" from \"wrote the wrong thing\". |\n\n## Arguments\n\nTaskset knobs are passed as `--env.taskset.<name>` (kebab-case) to `eval`, or as\n`--taskset.<name>` to `validate`; the list-valued `families` is easiest to set in a TOML\nconfig passed with `@ file.toml` (`[env.taskset] families = [\"repair\"]`).\n\n| Argument | Default | Meaning |\n|---|---|---|\n| `families` | `[\"repair\", \"reconcile\", \"merge\"]` | Families to cycle through by task index. |\n| `num-rows` | `60` | Rows per generated order table. The shipments table has half as many orders, a tenth of which get a second shipment row. |\n| `mess-rate` | `0.18` | Share of `units`, `unit_price` and `region` cells replaced by a missing-value marker. |\n| `seed` | `20260827` | Generator seed. All families draw from one generator, so changing any knob changes every later task, not only the family it concerns. |\n| `task.float-tolerance` | `0.001` | Absolute tolerance for every float comparison in the reward. |\n\n## Runtime\n\n- `NEEDS_CONTAINER = True`. Every task sets `image` and `workdir`, and `validate` refuses\n  the subprocess runtime, so `--runtime.type docker` (or `prime` / `modal`) is required.\n- **Image**: `amancevice/pandas:slim-3.0.5` — Python 3.13 on Debian slim with pandas 3.0.5\n  and numpy 2.5.2 preinstalled, published on Docker Hub for linux/amd64 and linux/arm64.\n  It is a community-maintained image, not one published by the pandas project, and the\n  tag is release-named rather than digest-pinned. The runtimes' default `python:3.11-slim`\n  ships no pandas, which is why the task declares one. `validate` runs\n  `python3 -c \"import pandas\"` inside the container, so a missing or substituted image\n  fails validation instead of every rollout.\n- **Files**: `setup` writes the input files (`raw.csv`, or `left.csv` and `right.csv`)\n  under `/workspace`; `finalize` reads the output file (`clean.csv` or `result.json`, at\n  most 4 MB) from the same directory.\n- **Network**: the task does not restrict egress; the runtime's own policy applies\n  (unrestricted by default). Nothing in the task needs the network.\n- **Timeouts and resources**: no per-task `timeout` or `resources` are set, so the\n  runtime and episode defaults apply. Tables are 60 rows by default, so the defaults are\n  ample; raise `--env.timeout.episode` if you scale `num-rows` far beyond that.\n\n## Dependencies\n\nThe package depends only on `verifiers>=0.3.1`; the reference implementations are\nstandard library (`csv`, `decimal`). pandas is needed only inside the sandbox and comes\nfrom the image above — nothing pandas-related is installed on the host. No secrets or\nenvironment variables are needed beyond the model provider's API key.\n\n## Run\n\n```bash\nuv run validate pandas-agentic-v1 -n 10 --runtime.type docker      # model-free: keys re-derive, pandas imports in the container\nuv run eval pandas-agentic-v1 -n 10 --model <model-id> \\\n  --env.agent.harness.id bash --env.agent.runtime.type docker\nuv run eval pandas-agentic-v1 -n 10 --model <model-id> \\\n  --env.agent.harness.id bash --env.agent.runtime.type docker \\\n  --env.taskset.num-rows 200 --env.taskset.mess-rate 0.3       # longer tables, more corruption\n```\n\nAlways pass `--env.agent.runtime.type`. Without it `eval` falls back to the Prime sandbox\nruntime, which provisions a remote, billed sandbox for every rollout.\n\n## Design notes\n\n**The reference is stdlib-only.** The agent will reach for pandas; if the grader used\npandas too, the two would share its missing-value, dtype-coercion, `groupby(dropna=...)`\nand merge-duplication conventions, and the environment would quietly stop testing whether\nthe agent handled them. The contract lives in the prompt, not in a library's defaults.\n\n**Answer keys re-derive from the shipped bytes.** `validate()` recomputes every key from\nthe exact file bodies the agent receives, so a generator bug fails validation instead of\nshipping a wrong key.\n\n**Half-up rounding, stated explicitly.** Python's `round()` and pandas' `.round()` are\nbanker's rounding and would send 12.345 to 12.34. The prompt promises the ordinary\nconvention and the reference implements it with `Decimal`.\n\n**Every reward ships with a test that tries to cheat it.** The offline tests in the source\nrepository assert that echoing the messy input, reordering rows, keeping a duplicate,\nleaving a sentinel in place, off-by-one counts, a phantom category, a banker's-rounded or\nunrounded mean, and a missing or malformed artifact all score zero; that every category\npresent in the data appears in the key even when all its prices are missing; that at least\none merge task in the benchmark slice actually multiplies joined rows; and they drive\n`setup` → `finalize` through a real subprocess runtime to cover the file plumbing.\n\n## Status\n\n| Check | Result |\n|---|---|\n| `uv run validate pandas-agentic-v1 -n 10 --runtime.type docker` | **10/10 valid**, 0 errors, 29 s wall clock, on the Docker runtime with the image above: keys re-derive and `import pandas` succeeds in every container. |\n| Offline tests and lint | 21 tests for this environment pass (`pytest tests/test_pandas_agentic.py`); `ruff check` and `ruff format --check` clean |\n| Model eval — `openai/gpt-5-mini`, 2026-09-12 | **1.000** `artifact_correct` at the defaults (`num_rows` 60, `mess_rate` 0.18) and again at `num_rows` 400, `mess_rate` 0.5; 10/10 tasks completed in both runs. Reproducible with `uv run eval pandas-agentic-v1 -n 10 --model openai/gpt-5-mini --env.agent.harness.id bash --env.agent.runtime.type docker` (add `--env.taskset.num-rows 400 --env.taskset.mess-rate 0.5` for the harder setting). |\n| Model eval — `openai/gpt-5-nano`, 2026-09-12 | **0.600** `artifact_correct`, `produced_output` 0.7 at the defaults, 10/10 completed: three tasks produced no usable artifact (`produced_output` 0) and one produced a wrong one. The environment separates the two models; gpt-5-mini saturates it at both settings. |\n| prime-rl training | **Not yet run.** |\n","encoding":"utf-8","truncated":false,"total_bytes":9541},"status":null}