{"data":{"kind":"file","path":"README.md","version_id":"jtvq5k84kykkl9kmy7ms865t","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5967,"modified_at":"2026-09-06T01:11:02.290000","content_hash":"460296fe68422061c93102a471f02e491b9ddf6821ab60312b8a0750f224d89f"},"entries":[],"content":"# secret-redaction\n\nWrite a redactor that removes credentials from developer text **without corrupting\nthe code around them.**\n\nThe model works in a container as a coding agent: it writes `redactor.py` exposing\n`scrub(text) -> str`, and is told to write its own test cases and run them before\nfinishing. Whatever file it leaves behind is graded against a **held-out** corpus it\nnever sees, scored on both sides at once:\n\n| metric | meaning |\n|---|---|\n| `wrote_file` | did the agent create `redactor.py` at all |\n| `caught` | fraction of secret samples whose secret no longer appears |\n| `intact` | fraction of benign samples returned byte-identical |\n| **reward** | **`caught x intact`** |\n\n## Why the reward is a product\n\nThis is the whole design.\n\nUnder a mean, `return \"[REDACTED]\"` scores **0.50** and `return text` scores **0.50**.\nBoth are trivial, neither redacts anything usefully, and both would out-earn a real\nfirst attempt at the problem. Under a product they score **0.00**, because each one\nzeroes a factor. The only way to score is to actually discriminate.\n\nMeasured, on the full tier:\n\n```\nstrategy              caught  intact  product   (mean)\n------------------------------------------------------\nnuke_everything         1.00    0.00     0.00     0.50\nidentity_noop           0.00    1.00     0.00     0.50\nempty_string            1.00    0.00     0.00     0.50\ncrashes                 0.00    0.00     0.00     0.00\nwrong_signature         0.00    0.00     0.00     0.00\nno_scrub_defined        0.00    0.00     0.00     0.00\n------------------------------------------------------\nreference               1.00    0.94     0.94     0.97\n```\n\n`tests/test_grader.py` asserts this in both directions on a stock interpreter — no\n`verifiers`, no GPU. It fails if any degenerate strategy scores above zero, if the\nreference drops below a 0.85 solvability floor, if the reference hits a perfect 1.00\n(a saturated corpus has nothing left to teach), **and** if the degenerate strategies\never stop scoring 0.50 under a mean — because then the corpus has drifted and the\nargument for the product reward has quietly stopped being true.\n\n## Why the corpus is adversarially paired\n\nA redactor cannot pass by matching surface form, because for nearly every secret\nshape there is a benign twin that looks the same to a careless pattern:\n\n| must be removed | must survive byte-identical |\n|---|---|\n| `sk_live_51QQWW...` | `settings.stripe_secret_key` |\n| `AWS_SECRET_ACCESS_KEY=wJalr...` | `os.environ[\"ANTHROPIC_API_KEY\"]` |\n| `203.0.113.47` | `127.0.0.1`, `0.0.0.0`, `172.18.0.4`, `192.168.1.50` |\n| `password=\"hunter2-actual-secret\"` | `password = \"your-password-here\"` |\n| `token = xoxb-1234...` | `token = secrets.token_urlsafe(32)` |\n| `/Users/jrivera/notes.md` | `/Users/Shared/config.json`, `jriveraX` |\n\nThree tiers of rising difficulty — `basic`, `adversarial`, `full` — configurable via\n`--env.tier`. Tasks draw held-out samples from the tier pool, so a model that\nhardcodes the prompt's examples scores near zero.\n\n## The two cases the reference still fails\n\nThe reference solution catches 28/28 secrets but corrupts 2 of 32 benign samples,\nand both failures are real production incidents rather than invented difficulty:\n\n```\nfour part semver     'bumped agp to 8.1.0.2'      -> 'bumped agp to [IP]'\nsvg path coordinates 'd=\"M8.4c.5.38.8.97.8 1.6\"'  -> 'd=\"M8.4c.[IP].8 1.6\"'\n```\n\nA dotted quad is not a reliable signal. The rule that lets `reportlab 4.0.9.1`\nsurvive is the same rule that lets a live server address print; the rule that\ncatches the address mangles SVG geometry. In the archive this corpus is drawn from,\nthe first mistake shipped two real server IPs into a printed book, and the second\nrewrote path data into `[IP-01]`. **That gap is the headroom this environment exists\nto train against.**\n\n## Provenance\n\nDistilled from a production redaction pipeline that gates a ~4,300-page published\narchive, where both sides genuinely matter: a leak publishes a credential, and a\nfalse positive corrupts source code that has to remain runnable. The upstream suite\nholds 19 must-redact and 20 must-survive cases with a disabled-redactor control.\n\nEvery secret in this corpus is synthetic. No credential here was ever live.\n\n## Sandboxing\n\nThe candidate redactor is model-written code that gets executed, so each task\ncarries its own limits rather than trusting the default: `network_block=[\"*\"]`,\na 60s scoring ceiling, 1 CPU, 0.5 GB, 1 GB disk. A redactor that tries to reach the\nnetwork, allocate without bound, or backtrack forever fails closed and scores zero\ninstead of hanging the run.\n\n## Verification\n\nTwo suites, deliberately split by what they need:\n\n```bash\npython3 tests/test_grader.py     # no dependencies, any interpreter\npython3 tests/test_taskset.py    # needs verifiers; skips cleanly without it\npython3 tests/test_runtime.py    # needs verifiers + docker; skips cleanly without\n```\n\n`test_grader.py` proves the reward is sound (see above). `test_taskset.py` proves\nthe wiring is real against the installed `verifiers`. `test_runtime.py` proves\ngrading works **inside a real container** - the path production actually uses - and\nthat a missing or broken `redactor.py` degrades to a clean zero rather than crashing — every tier loads, held-out\ndraws differ per task, the sandbox limits are attached to *every* task rather than\njust declared, no graded sample appears verbatim in the prompt, and an unknown tier\nraises instead of dying on a `KeyError` deep inside `load()`.\n\nBoth currently pass — 144 tasks across 4 tiers, verified against `verifiers` 0.3.1\non Python 3.12.13.\n\n## Usage\n\n```bash\nuv pip install -e .\nuv run eval secret_redaction -n 3\nuv run eval secret_redaction --env.taskset.tier basic\nuv run eval secret_redaction --env.agent.runtime.type docker   # local, free\n```\n\nNote `eval`, not `vf-eval` - the latter is the legacy evaluator and cannot load a\nv1 taskset. The module id uses an underscore.\n","encoding":"utf-8","truncated":false,"total_bytes":5967},"status":null}