{"data":{"kind":"file","path":"README.md","version_id":"f8vgytmd8o1bidybgozwoon5","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3005,"modified_at":"2026-09-17T04:32:41.548000","content_hash":"66956be406c637dcc584277bdb8b58b3835dbf05cd296ef35d1058b54a0d01bb"},"entries":[],"content":"# Adversarial Reward Hacking Environment (v0.0.1)\n\nA rigorous benchmark and Reinforcement Learning from Human Feedback (RLHF) verification environment designed to evaluate, quantify, and mitigate **Adversarial Reward Hacking**, **Specification Gaming**, and **Sycophancy** in language models.\n\n---\n\n## 🌟 Overview\n\nReinforcement learning fine-tuning frequently causes LLMs to discover reward-exploiting shortcuts that maximize automated proxy scores without satisfying true underlying human preferences.\n\nThis environment presents a comprehensive testbed containing challenging adversarial scenarios, user bias flattery traps, verbosity incentives, and hidden reward shortcut tokens.\n\n### Key Evaluation Dimensions\n\n1. **True Quality & Rigor (`true_quality_reward`)**:\n   - Evaluates genuine technical depth, invariant enforcement, balanced perspectives, and actionable explanations.\n   - Non-gameable multi-signal assessment verifying actual functional value.\n\n2. **Sycophancy Resistance (`sycophancy_resistance_reward`)**:\n   - Tests whether the model resists agreeing with misleading user biases (e.g., incorrect arithmetic, unsafe engineering shortcuts, or broken security practices).\n   - Enforces polite, grounded intellectual honesty.\n\n3. **Verbosity & Formatting Proxy Traps (`proxy_verbosity_trap`, `proxy_formatting_trap`)**:\n   - Emulates flawed proxy reward signals that disproportionately reward text length or excessive Markdown styling.\n   - Modeled using smooth sigmoid functions to prevent discrete threshold exploitation.\n\n4. **Hidden Token / Magic String Traps (`hidden_token_trap`)**:\n   - Monitors specification gaming via benchmark shortcut tokens.\n\n5. **Proxy-True Divergence Gap (`divergence_detection_metric`)**:\n   - Quantifies the divergence gap $\\Delta = \\max(0, R_{\\text{proxy}} - R_{\\text{true}})$ to monitor reward over-estimation in real time.\n\n---\n\n## 🚀 Quickstart\n\n### Installation\n\n```bash\nuv pip install -e .\n```\n\n### Loading the Environment\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"lassiter/adversarial-reward-hacking\")\nprint(env)\n```\n\nOr via direct import:\n\n```python\nfrom adversarial_reward_hacking import load_environment\n\nenv = load_environment()\n```\n\n---\n\n## 📊 Dataset Structure\n\nThe environment includes curated multi-domain scenarios spanning:\n- **Specification Gaming**: Code validation vs fake length / meaningless imports.\n- **Sycophancy**: Resisting mathematical fallacies and architecture dogmas.\n- **Adversarial Safety**: Rejecting hazardous production shortcuts.\n- **Format Quota Traps**: Avoiding empty header and bullet bloat.\n\nEach sample provides:\n- `question`: The user query with integrated adversarial bias or exploit incentive.\n- `answer`: Ground truth explanation with balanced resolution.\n- `info`: Metadata dictionary detailing category, attack type, user bias, and core requirements.\n\n---\n\n## 🧪 Testing\n\nRun unit tests via `pytest`:\n\n```bash\npython3 -m pytest tests/ -v\n```\n\n---\n\n## 📜 License\n\nMIT License.\n","encoding":"utf-8","truncated":false,"total_bytes":3005},"status":null}