{"data":{"kind":"file","path":"README.md","version_id":"sih4mij5b7ym70zpzto3q2tp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8564,"modified_at":"2026-11-02T14:41:53.782000","content_hash":"2470015f313f027921b5d8c5f0724aedbacb2fe6c08103adabd849263506e53a"},"entries":[],"content":"# DataClean-RL\n\n**A deterministic, preservation-aware data cleaning environment for Prime Intellect's Environments Hub (Verifiers v1).**\n\nDataClean-RL trains or evaluates an AI agent on a *real action* rather than a natural-language claim: the model must return a JSON program of approved cleaning operations, which the environment executes against a corrupted structured table. The resulting table is compared with private synthetic ground truth, with penalties for missing fixes, overwriting already-correct values and dropping rows.\n\n> **Validation scope:** the 23 local tests pass with Verifiers 0.4.0 installed, and `vf-eval dataclean-rl -n 1 --dry-run --no-push` resolves successfully. This does not establish live-model performance, hosted CI success, or bounty acceptance. The historical Prime Intellect bounty spreadsheet explicitly says its bounty program has closed: <https://docs.google.com/spreadsheets/d/13UDfRDjgIZXsMI2s9-Lmn8KSMMsgk2_zsfju6cx_pNU/edit>.\n\n## Why this is useful\n\nGeneric cleaning benchmarks often reward over-aggressive deletion or exact output strings without proving that important data survived. DataClean-RL makes false cleaning costly and defines an executable, reproducible action space:\n\n- **Three task families:** customer CRM, order exports and warehouse inventory; no real customer information or external data required.\n- **Messy data:** names with international characters, casing and whitespace, phone formatting, date formats, amount formatting, category aliases, null-like strings, duplicates and zero/negative values that must be preserved.\n- **Per-task unpredictable codes:** each task adds two unique, explicitly documented export aliases and a variable US/EU date convention. One hard-coded fixed cleaning plan cannot solve all tasks.\n- **Ground truth separated from agent input:** clean records created first, independently corrupted later. Gold is never in the prompt.\n- **Deterministic and fully auditable reward:** repair recall (60%), preservation accuracy (20%), row integrity (20%). Valid actions are strictly interpreted. No model grader or external data leaks.\n- **Split isolation:** separate deterministic RNG streams for `train`, `eval` and `test`, with seed-controlled generation and no network dependencies.\n- **Safety:** deny-by-default JSON DSL, whitelist of operations/columns, no Python or SQL execution, bounded plan length; `record_id`, `customer_id` and `notes` cannot be modified through DSL operations.\n\n## Environment contract\n\nThe agent observes one policy and a JSON table with 10–18 unique rows plus 0–2 duplicates. It returns **only**:\n\n```json\n{\n  \"operations\": [\n    {\"op\": \"strip\", \"columns\": [\"name\", \"email\", \"phone\", \"segment\", \"signup_date\"]},\n    {\"op\": \"replace_missing\", \"columns\": [\"email\", \"phone\"], \"tokens\": [\"N/A\", \"null\", \"-\", \"\"]},\n    {\"op\": \"lower\", \"columns\": [\"email\"]},\n    {\"op\": \"phone_us\", \"columns\": [\"phone\"]},\n    {\"op\": \"date_us\", \"columns\": [\"signup_date\"]},\n    {\"op\": \"map\", \"column\": \"segment\", \"mapping\": {\"VIP\": \"vip\", \"Std\": \"standard\"}},\n    {\"op\": \"dedupe\", \"keys\": [\"record_id\"]}\n  ]\n}\n```\n\nThis is an **illustrative partial plan**, not a solution to a whole task: each instance has additional mappings, including two one-off aliases and the US/EU date convention provided in the case policy. Allowed operations: `strip`, `lower`, `upper`, `map`, `replace_missing`, `phone_us`, `date_us`, `date_eu`, `decimal2`, `integer`, `dedupe`. They are applied in order. No code, tools, external API or disk access is executed in the scorer.\n\n### Scoring\n\nFor each reference `record_id`, cells are divided into those that were originally corrupted and those that were already correct. We align by primary key (row order does not matter). Duplicates, missing rows and invented IDs affect structure. The reward is:\n\n`0.60 × repair_recall + 0.20 × preservation_accuracy + 0.20 × row_integrity`\n\n- **Repair recall** = corrected affected cells ÷ all originally corrupted cells. Unrepaired missing values fail.\n- **Preservation accuracy** = unchanged ground-truth cells still correct ÷ all originally correct cells. Unwarranted edits fail.\n- **Row integrity** = 1 − (extra row occurrences + missing row occurrences) ÷ max(1, submitted + expected rows).\n- **Invalid programs** receive 0, including code execution requests, edits to protected fields, wrong operation schemas and unsupported duplicate keys.\n\nIf a task has no changed cells, repair recall is defined as 1.0. A fully correct table earns exactly 1.0; doing nothing typically earns around 0.4 on the synthetic sample (see demo), not a high reward merely from intact cells.\n\n### Reproducibility and leakage\n\nThe generator uses a stable per-split arithmetic seed, never Python's randomized `hash()`. The per-task rules are visible to the agent, and the gold records exist only in evaluation fields. Because this is open-source synthetic generation, we **do not claim cryptographically private hidden test data**. A stronger blind holdout should be generated by an independent evaluator with secret seeds and new corruption templates; see `docs/REVIEW_CHECKLIST.md`.\n\n## Installation\n\nRequires Python 3.11–3.14 and **Verifiers 0.4.x**, with the Prime CLI / API key only for actual Prime-hosted model runs.\n\n```bash\n# From the unzipped project directory\nuv venv\nuv pip install -e \".[dev]\"\n\n# Tests of the cleaning engine and integration (integration runs if Verifiers installed)\nuv run pytest -q\n\n# Offline deterministic baseline, no API key or LLM endpoint\nuv run python scripts/demo.py\n```\n\n### Local evaluation via Verifiers\n\n```bash\n# In a Prime Lab workspace, install this package from its directory:\nuv pip install -e /path/to/dataclean_rl_environment\n\n# Smoke-test with your configured model and API key; first verify CLI availability:\nuv run vf-validate dataclean-rl\nuv run vf-eval dataclean-rl -n 5 -r 1 --model <YOUR_MODEL_ID>\n```\n\nThe actual `vf-eval` CLI argument names can vary across Prime CLI releases; consult `uv run vf-eval --help` if needed. When using `prime eval run`, follow the installed Prime CLI's help. **These hosted commands have not been run in this artifact creation environment.**\n\nTaskset config is `DataCleanTasksetConfig(split=\"train\", seed=42, num_tasks=90)`; typical evaluation config uses `split=\"eval\"` with 30+ tasks and multiple rollouts per task. The package exports one `DataCleanRLTaskset` class via `__all__` and uses `verifiers.v1` rather than removed legacy interfaces.\n\n## Benchmark methodology (submission readiness)\n\nTo claim quality, produce a public evaluation record with:\n\n1. `uv run pytest -q` and `uv run vf-validate dataclean-rl` passing with Verifiers 0.4.x.\n2. At least 100 held-out tasks, 3 rollouts per task and fixed model/temperature/token budget.\n3. A no-op baseline, the deterministic oracle, a heuristic baseline, at least two external agent models, and confidence intervals for mean reward.\n4. Per-domain breakdown and rates of invalid JSON, accidental row deletion, preservation violations, and exact full-table success.\n5. A public model trace sample with customer information guaranteed synthetic and a reproducibility file documenting exact versions, seed, split and prompts.\n6. External review of v1 package compatibility, gold leakage, eval distribution, and reward exploits.\n\nThese are **proposed acceptance gates**, not Prime Intellect's confirmed bounty requirements.\n\n## Project structure\n\n```text\ndataclean_rl/\n  generator.py       Deterministic tasks + per-task aliases\n  engine.py          Safe action interpreter + reward\n  taskset.py         Verifiers v1 integration\n  __init__.py        Package entrypoint\nscripts/demo.py      Local oracle/no-op demonstrations\nscripts/benchmark_local.py   Offline CSV + summary baselines\ntests/               Regression and security tests\ndocs/BOUNTY_PROPOSAL.md\n docs/REVIEW_CHECKLIST.md\n.github/workflows/ci.yml\n```\n\n## License and contribution\n\nApache-2.0. See `LICENSE`. Do not submit this as a claimed bounty without written scope or a current Prime Intellect bounty assigned to you. You are welcome to contribute additional domain families, adversarial examples and independent held-out datasets while keeping the scoring API backwards compatible.\n\n## Offline benchmark report\n\nSee [`docs/LOCAL_BASELINES.md`](docs/LOCAL_BASELINES.md) and [`docs/local_baselines.csv`](docs/local_baselines.csv) for model-free no-op, generic-script and reference-policy results. These verify that task-specific instructions matter, but do **not** establish performance of any real AI agent.\n","encoding":"utf-8","truncated":false,"total_bytes":8564},"status":null}