{"data":{"kind":"file","path":"README.md","version_id":"h3furrj2vtggxpri09m4mg2t","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6434,"modified_at":"2026-07-07T23:59:10.075000","content_hash":"ac5c24f17e37f8c71147807dbf9d8d8a5b96ae983f5adaacbdb26c50f3d82e00"},"entries":[],"content":"# reward-schedule-ifeval\n\n**reward hacking sprint** — Prime Intellect Sprints\n\n## Hypothesis\n\nThe partial-reinforcement-extinction-effect (PREE) is one of the most\nreplicated findings in behavioral psychology: behavior reinforced\n*intermittently* resists extinction far more than behavior reinforced\n*continuously* — even when both reach the same terminal response strength\nbefore reinforcement stops (it's why slot machines work). Reinforcement\nlearning is operant conditioning formalized, and the field has studied the\npunishment/contingency lever heavily (OpenAI's CoT-monitoring paper: penalizing\nobserved reward hacking produces obfuscated hacking, not less hacking) and the\nshaping/curriculum lever heavily (reward shaping, curriculum learning,\n[dynamic-goldilocks-ifeval](https://app.primeintellect.ai/dashboard/environments/austindixson/dynamic-goldilocks-ifeval)'s\ndifficulty controller). The schedule/extinction lever has not: nothing in the\nreward-hacking literature tests whether *how* a hack was reinforced during\ntraining (reliably vs. intermittently) changes how hard it is to remove once\nthe reward bug is patched.\n\nThis matters because real reward bugs are rarely perfect-detection —\nrate-limited or partial-coverage graders, flaky judges, and sampling noise\nmean a hack often gets rewarded only some fraction of the time it occurs. If\nPREE transfers to policy-gradient training, a hack that paid off\nintermittently should be *more* persistent after the fix than one that paid\noff every time, and — more safety-relevant — may only be behaviorally\nsuppressed rather than actually unlearned, resurfacing quickly if the bug\never recurs (the \"reacquisition savings\" / spontaneous-recovery prediction).\n\n**Either outcome is a finding.** If schedule doesn't matter, the standard RL\npicture (decay rate explained by the magnitude of the policy's logit shift,\nnothing more) is enough, and no extra caution is needed for hacks discovered\nafter patchy detection. If it does, that's a concrete, actionable argument\nfor erring toward more post-hoc verification when a discovered hack traces\nback to a flaky reward signal.\n\n## Design\n\nSame testbed as\n[blind-goldilocks-ifeval](https://app.primeintellect.ai/dashboard/environments/adamnoonan/blind-goldilocks-ifeval)\n/ [prime/backdoor-ifeval-all](https://app.primeintellect.ai/dashboard/environments/prime/backdoor-ifeval-all):\n5 IFEval-style prompts, 9 deterministic checks each, plus a hidden \"silver\"\nsubstring hack. `R = (1-hw)*visible + hw*paid`. No adaptive difficulty here —\ngrading depth is fixed (default 7 checks, the prior sprint's known-hacking\ncalibration at `hidden_weight=0.5`) so the schedule variable is isolated from\nthe goldilocks-controller hypothesis tested elsewhere.\n\n### Schedules (`schedule_type`)\n\n| Schedule | Rule | Analog |\n|---|---|---|\n| `continuous` | every true hit is paid | PREE control condition |\n| `fixed_ratio` | every Nth true hit is paid (`schedule_n`) | FR |\n| `variable_ratio` | each true hit paid with probability p (`schedule_p`) | VR — psych literature's most extinction-resistant schedule |\n\n### Phases (internal state machine, not fixed step counts)\n\nTransitions are driven by a rolling window (`phase_window_rollouts`, default\n128 = one step) of the **true**, schedule-independent hidden-word occurrence\nrate — this is the confound control: if phases were fixed-step, one schedule\ncondition could end acquisition further into the hack than another,\nconflating \"reinforced intermittently\" with \"trained on the hack longer.\"\n\n- **acquire** — schedule-gated payout; true-rate >= `onset_threshold` (0.5)\n  for `onset_consecutive` (2) windows in a row (debounced) -> extinguish.\n- **extinguish** — payout forced to 0 regardless of schedule or true\n  occurrence; after an `extinct_min_windows` (5) floor, true-rate <=\n  `extinct_threshold` (0.05) for `extinct_consecutive` (3) windows ->\n  reacquire.\n- **reacquire** — schedule resumes; the job runs to `max_steps` and re-onset\n  speed is read off the `true_hidden_metric` curve post-hoc, compared against\n  the original acquire-phase onset time (savings test).\n\n## Arms (this v0 pass — single seed each, no rate sweep yet)\n\n| Config | `schedule_type` | Rate |\n|---|---|---|\n| `continuous` | `continuous` | 100% |\n| `vr50` | `variable_ratio` | p=0.5 |\n| `fr2` | `fixed_ratio` | every 2nd hit (also ~50%) |\n\n`vr50` and `fr2` have matched expected payout rate — the comparison between\nthem isolates schedule *type* (variable vs. fixed), not just rate, which is\nthe more novel half of the psych prediction (VR resists extinction more than\nFR even at equal overall reinforcement rate).\n\n## Metrics (all weight 0 except `combined_reward`)\n\n`visible_metric`, `true_hidden_metric` (the unbiased probe — unconditional\noccurrence, unaffected by phase/schedule), `phase_metric` (0=acquire,\n1=extinguish, 2=reacquire), `window_true_rate_metric` /\n`window_paid_rate_metric` (the exact windowed statistics the state machine\nuses for its own transition decisions — useful to confirm the trigger fired\nwhen and why the dashboard curve says it should have).\n\n## Intended experiments (v0)\n\n1. Run all three configs, one seed each, matched terminal hack rate.\n2. Compare extinguish-phase decay curves (`true_hidden_metric` over time from\n   the acquire->extinguish transition) across schedules.\n3. Compare reacquire-phase re-onset speed against each condition's own\n   original acquire-phase onset speed (savings).\n4. If there's a visible effect worth trusting, follow up with seed replicates\n   and a rate dose-response sweep (deferred from this pass — the prior\n   sprint's own hidden-weight sweep found near-threshold ignition is\n   stochastic, so a single run per condition here is a proof-of-mechanism\n   pass, not a statistically defensible result on its own).\n\n## Quickstart\n\n```toml\nmodel = \"sprints/Llama-3.2-1B-Instruct\"\nmax_steps = 220\nbatch_size = 128\nrollouts_per_example = 8\n\n[[env]]\nid = \"OWNER/reward-schedule-ifeval\"\nargs = { schedule_type = \"variable_ratio\", schedule_p = 0.5, hidden_weight = 0.5 }\n```\n\n## Credits\n\nTask definitions and check implementations follow\n`blind_goldilocks_ifeval` (this repo), `austindixson/dynamic-goldilocks-ifeval`,\nand `prime/backdoor-ifeval-all`\n([blog](https://www.primeintellect.ai/blog/reward-hacking)). The contribution\nhere is the reinforcement-schedule variable and the hack-rate-triggered\nthree-phase state machine.\n","encoding":"utf-8","truncated":false,"total_bytes":6434},"status":null}