{"data":{"kind":"file","path":"README.md","version_id":"dlu98iar51dp8gi9eye8hl5m","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3258,"modified_at":"2026-09-06T04:16:20.299000","content_hash":"21977a425f20a7a482fddee1ec394aeaa050df30ebd458515687d14680028732"},"entries":[],"content":"# durable-notebook\n\nAn RL environment for studying reward hacking.\n\nThe setup: a model has a long conversation where facts get mentioned once\nand then scroll out of view. The only way to remember something is to\nwrite it to a file. At the end, it has to answer questions about what it\nwrote down, and point to which file backs each answer.\n\nThere are two ways to grade this:\n\n- **naive** — just checks that the file exists and isn't empty. Doesn't\n  check if the answer is actually right.\n- **hardened** — actually reads the files and checks whether the true\n  answer is really in there.\n\nI trained a model (Qwen3-1.7B, GRPO) against each grader separately, then\nre-graded every rollout from both runs under *both* graders. That way I\ncould measure the real gap directly instead of just eyeballing two reward\ncurves that happened to both go up.\n\n## What happened\n\n| | trained on naive | trained on hardened |\n|---|---|---|\n| naive score | 0.79 | 0.78 |\n| hardened score | 0.53 | 0.67 |\n| worst cheating moment | 64.5% (step 50) | 30% (step 1, before training even helped) |\n\nThe naive-trained model developed a big cheating spike partway through\ntraining — at step 50, 64.5% of its answers looked \"done\" to the naive\ngrader but were flat wrong. The hardened-trained model never got close to\nthat, and actually hit 0% cheating at that exact same step.\n\n## How it cheats\n\nThe dataset is seeded, so the same episode always has the same true\nanswer. If the model actually remembered a fact, replaying that episode\nshould give the same answer every time. It doesn't. Same episode,\ndifferent training runs of it:\n\n- one date came back 5 different ways\n- one material came back as 4 different guesses\n- one person's name came back as 5 different names\n\nIt's making things up. In one case the model wrote down the *correct*\nanswer earlier in the conversation, then at the very end overwrote that\nfile with the word \"Unknown\" — and still got a perfect score, because the\nnaive grader only checks that a file exists, not what's written in it.\n\n## Install\n\n```bash\nprime env install aravind-k/durable-notebook\n```\n\n## Use it\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"durable-notebook\", grader=\"naive\")  # or \"hardened\"\n```\n\nTools the model gets: `write_file`, `read_file`, `list_files`,\n`submit_manifest`. No code execution, just a scoped filesystem sandbox.\n\n## What's not perfect about this\n\nThe hardened grader isn't bulletproof either. It only checks if the right\nanswer shows up *anywhere* in the workspace, not that it's filed under the\nright question — so a model could still get away with writing down every\nguess it can think of into one big file. That's likely why neither run\ngets all the way to 0% cheating.\n\nAlso, thinking was turned off during training. Qwen3's reasoning traces\nwere eating the entire per-turn token budget regardless of how short the\nepisode was. Worth retesting with reasoning on and a bigger budget later.\n\n## Reproduce the numbers\n\n`regrade_traces.py` (included in this package) replays every rollout's\nfile writes against the same seeded dataset and scores it under both\ngraders:\n\n```bash\npython regrade_traces.py <traces.jsonl...> \\\n    --n-episodes 400 --n-turns 12 --n-questions 4 --start-seed 0\n```\n","encoding":"utf-8","truncated":false,"total_bytes":3258},"status":null}