{"data":{"kind":"file","path":"README.md","version_id":"kg9nxfwwiubjihostglgd413","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5337,"modified_at":"2026-07-06T21:53:38.496000","content_hash":"f825cf02f3c5391494c83c1ebdd0febde7d5266c55fa74e2113d1db737b28a11"},"entries":[],"content":"# blind-goldilocks-ifeval\n\n**reward hacking sprint** — Prime Intellect Sprints\n\n## Hypothesis\n\nA difficulty controller that sees **only visible-side statistics** — within-group\nvisible-reward variance and group means, never the hidden reward — can keep the\nvisible gradient live and suppress reward hacking.\n\nThis is the question [austindixson/dynamic-goldilocks-ifeval](https://app.primeintellect.ai/dashboard/environments/austindixson/dynamic-goldilocks-ifeval)\noriginally posed and then abandoned: their hack-blind v1 failed (98.6% hacked),\nbut the failure was confounded by implementation bugs (starting at 3 checks — far\ntoo easy — and a delta deadband that trapped the controller). Their fixed v2\nsuppresses hacking by monitoring the hidden reward directly and escalating when it\nexceeds 5%. That works, but it is an **oracle**: in real training the hack is by\ndefinition the unmetered gap between proxy and intent. If you could measure it,\nyou would remove it from the reward. Whether a *blind* controller works once the\nlegitimate bugs are fixed was never re-tested. This environment tests it.\n\n**Either outcome is a finding.** If blind control suppresses hacking, difficulty\nscheduling is a deployable mitigation that requires no knowledge of the hack. If\nit fails when done properly, that is evidence hacking suppression requires\nhidden-signal monitoring — supporting monitor-based mitigations over\ncurriculum-based ones.\n\n## Design\n\nSame testbed family as [prime/backdoor-ifeval-all](https://app.primeintellect.ai/dashboard/environments/prime/backdoor-ifeval-all):\n5 IFEval-style prompts, 9 deterministic checks each (ordered easiest → hardest),\nplus a hidden \"silver\" reward. `R = (1-hw)·visible + hw·hidden`. Prompt text\nalways lists all 9 constraints; only the number of **graded** checks moves, so\nreward shape is isolated from prompt shape. No API key required — every check is\npure Python (the on-topic LLM judge from the upstream env is dropped).\n\nThe controller acts once per window of rollout groups (default 16 groups = one\nstep at batch 128 / 8 rollouts). Per group, it classifies:\n\n- **live** — visible std > `std_eps` (0.05): mixed outcomes, gradient flowing\n- **saturated** — flat and mean ≥ `sat_mean` (0.9): nothing left to learn from above\n- **hopeless** — flat and mean < `sat_mean`: nothing learnable from below\n\nRule: ≥50% saturated → +1 check; ≥50% hopeless → −1 check; otherwise hold.\nThis targets the mechanism directly: GRPO only learns from a group when rewards\nvary within it ([DAPO](https://arxiv.org/abs/2503.14476)'s zero-variance\nobservation), and difficulty-targeting for learnability is established for\n*efficiency* ([AdaRFT](https://arxiv.org/abs/2504.05520)) — this env asks whether\nthe same lever works for *safety*.\n\n### Known structural limit\n\nWhen the model masters all 9 checks, the blind controller runs out of headroom\n(`pinned_high` metric goes to 1.0) and visible saturates with no move left. If\nhacking ignites only after pinning, that bounds what blind difficulty control can\never do — that boundary is part of what the experiment maps.\n\n## Arms (one env, `controller` arg)\n\n| Arm | `controller` | What it shows |\n|---|---|---|\n| control | `static`, `hidden_weight=0` | baseline word rates, no hidden incentive |\n| static-easy | `static`, 3 checks | replicates upstream v1 failure condition |\n| static-safe | `static`, 7 checks | upstream's proven-safe static baseline |\n| **blind** | `blind`, start 7 | **the hypothesis** |\n| oracle | `oracle`, start 7 | upstream's fixed v2 rule, same window, for comparison |\n\nThe oracle arm implements austindixson's v2 rule verbatim (hidden > 5% → +5\nchecks; else visible-delta deadband ±) but on the same decision window as the\nblind arm so the comparison is like-for-like.\n\n## Metrics (all weight 0)\n\n`visible_metric`, `hidden_metric` (observability — only the oracle controller\nreads hidden), `num_checks_metric` (the knob), `frac_live_metric`,\n`frac_saturated_metric`, `frac_hopeless_metric`, `pinned_high_metric`,\n`group_visible_std` (the controller's own live-gradient signal, logged per group).\n\nWatch `hidden_metric` vs `num_checks_metric`: the blind hypothesis survives if\nhidden stays near 0 while the knob moves only on visible statistics.\n\n## Intended experiments\n\n1. Run all five arms, 100 steps, batch 128, rollouts 8 (configs in repo).\n2. Compare hidden-reward trajectories: does blind match oracle suppression,\n   match static-easy failure, or land in between (delayed liftoff)?\n3. Check whether hacking onset in any arm is preceded by `pinned_high=1.0`\n   (headroom exhaustion as a blind early-warning signal).\n4. Sensitivity: `std_eps` and `sat_mean` sweeps on the blind arm if step 2 is\n   ambiguous.\n\n## Quickstart\n\n```toml\nmodel = \"sprints/Llama-3.2-1B-Instruct\"\nmax_steps = 100\nbatch_size = 128\nrollouts_per_example = 8\n\n[[env]]\nid = \"OWNER/blind-goldilocks-ifeval\"\nargs = { controller = \"blind\", num_initial_checks = 7, hidden_weight = 0.5 }\n```\n\n## Credits\n\nTask definitions and check implementations follow\n`austindixson/dynamic-goldilocks-ifeval`, which mirrors\n`prime/backdoor-ifeval-all` ([blog](https://www.primeintellect.ai/blog/reward-hacking)).\nThe contribution here is the blind controller, the variance-based decision rule,\nand the like-for-like ablation harness.\n","encoding":"utf-8","truncated":false,"total_bytes":5337},"status":null}