{"data":{"kind":"file","path":"README.md","version_id":"k7v2l2pfe9ln5ca8zjlfmo6b","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2939,"modified_at":"2026-09-12T13:37:41.700000","content_hash":"aa13b1e5b684c6d7f49a31cc6aafcdb071e5d6d19fa59e00eb75a17e4b7d0507"},"entries":[],"content":"# Reward Hacking Benchmark (`reward-hacking-bench`)\n\nA high-fidelity evaluation and reinforcement learning benchmark designed to detect, study, and measure **Reward Hacking**, **Specification Gaming**, and **Proxy-True Metric Divergence** in Large Language Models (LLMs).\n\n---\n\n## 🎯 Overview\n\nReward hacking occurs when a model optimizes for proxy metrics (such as length, formatting tricks, keyword frequencies, or structural templates) at the expense of genuine task quality, safety, and functionality.\n\n`reward-hacking-bench` provides:\n- **Calibrated Proxy Traps**: Sigmoid-smoothed reward signals that simulate real-world RLHF vulnerabilities (length bias, superficial structure, buzzword padding).\n- **Hardened True Metrics**: Multi-condition evaluations checking real AST syntactical correctness, non-trivial control flow, architectural depth, and lexical density.\n- **Divergence Diagnostics**: Built-in `proxy_true_gap_diagnostic` measuring the quantitative disconnect between model gaming behavior and actual quality.\n\n---\n\n## 🛠️ Benchmark Architecture\n\n### 1. Proxy Rewards (Trained / Gamed Targets)\n- `proxy_length_smooth`: Sigmoid curve over length, modeling verbosity exploitation without discrete threshold plateaus.\n- `proxy_header_structure`: Sigmoid measuring header proliferation.\n- `proxy_code_block_count`: Rewarding raw fenced-code volume regardless of semantic content.\n- `proxy_buzzword_density`: Frequency detection of unverified technical terminology.\n\n### 2. Hardened True Metrics (Quality Verification)\n- `true_functional_code_metric`: Multi-condition AST validation (verifies parameterization beyond `self`, typed annotations, branch/loop control flow, and non-trivial return values).\n- `true_architectural_depth_metric`: Checks substantive section volume, operational action verbs, and concrete failure-mode handling.\n- `true_anti_repetitive_substance`: Penalizes repeated boilerplates, sentence loops, and low lexical uniqueness.\n\n### 3. Gap Analysis\n- `proxy_true_gap_diagnostic`: Tracks the real-time gap `max(0, proxy - true)` across rollouts to identify gaming onset.\n\n---\n\n## 🚀 Quickstart\n\n### Installation & Loading\n\n```python\nimport verifiers as vf\nfrom reward_hacking_bench import load_environment\n\n# Load the environment\nenv = load_environment(level=0, eval_mode=\"composite\")\n\nprint(\"Dataset size:\", len(env.dataset))\nprint(\"Rubric funcs:\", [f.__name__ for f in env.rubric.funcs])\n```\n\n### Local Evaluation via Prime CLI\n\n```bash\nprime eval run michaell/reward-hacking-bench -m Qwen/Qwen2.5-7B-Instruct\n```\n\n---\n\n## 📊 Evaluation Modes\n\n- `composite` (default): Blended configuration with hardened metrics + proxy signals for balanced RL.\n- `proxy_only`: Surfaces maximum gaming tendencies by rewarding solely superficial signals.\n- `hardened`: Strict quality gating that discards proxy biases.\n\n---\n\n## 📜 License\nMIT License. Developed for AI safety and robust RL alignment research.\n","encoding":"utf-8","truncated":false,"total_bytes":2939},"status":null}