{"data":{"kind":"file","path":"README.md","version_id":"se3q6mt3m10p9qzh0ca1q7w1","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3212,"modified_at":"2026-08-31T03:48:43.540000","content_hash":"2e082a202e67a6af908f1133e9fb6e4d1bbfcee1eda6c3257f5a37ddd2857260"},"entries":[],"content":"# Reinforcement Learning Hyperparameter Tuner (v0.1)\n\nAn environment that evaluates an LLM's ability to recommend optimal hyperparameters for reinforcement learning algorithms across diverse training scenarios.\n\n## Overview\n\nThis environment presents realistic RL training scenarios where the model must:\n1. Analyze the characteristics of the task (observation/action space, reward structure, episode length)\n2. Select appropriate algorithm and hyperparameter configurations\n3. Justify choices with domain-specific reasoning\n4. Handle edge cases like sparse rewards, continuous control, and multi-agent settings\n\n## Scenarios\n\nThe environment covers 6 RL algorithm families across 20+ distinct scenarios:\n\n| Algorithm | Use Case | Key Hyperparameters |\n|-----------|----------|-------------------|\n| PPO | Continuous control, robotics | clip_range, gae_lambda, n_epochs, ent_coef |\n| SAC | Sample-efficient continuous control | alpha, tau, batch_size, learning_rate |\n| DQN/Double-DQN | Discrete action spaces | epsilon_decay, target_update, buffer_size |\n| A2C/A3C | Parallel training | n_steps, vf_coef, max_grad_norm |\n| TD3 | Deterministic continuous policies | policy_delay, noise_clip, target_policy_noise |\n| REINFORCE | Simple policy gradient | gamma, learning_rate, baseline |\n\n## Reward Functions\n\n- **algorithm_match** (weight 0.20): Correct algorithm selection for the scenario\n- **hyperparameter_validity** (weight 0.25): Values within valid ranges and internally consistent\n- **reasoning_quality** (weight 0.25): Depth and correctness of justification\n- **domain_knowledge** (weight 0.15): References to relevant RL concepts and best practices\n- **edge_case_awareness** (weight 0.15): Handling of tricky scenarios (sparse rewards, reward hacking, etc.)\n\n## Difficulty Levels\n\n- **Level 0**: Standard scenarios with clear algorithm fit (e.g., CartPole + DQN)\n- **Level 1**: Ambiguous scenarios requiring nuanced choices (e.g., sparse reward + continuous control)\n- **Level 2**: Multi-objective or constrained scenarios (e.g., safety constraints, sample budget limits)\n- **Level 3**: Novel or hybrid scenarios requiring creative hyperparameter combinations\n\n## Example Prompt\n\n```\nYou are an RL hyperparameter tuning expert. Given the following training scenario,\nrecommend the optimal algorithm and hyperparameters.\n\nScenario: Robotic manipulation task with 7-DOF arm, dense shaped reward,\n200-step episodes, continuous action space [-1, 1]^7, 30-dimensional observation.\nTraining budget: 1M steps. Sample efficiency matters.\n\nProvide your recommendation as a structured configuration with justification.\n```\n\n## Evaluation\n\nThe environment uses multi-dimensional scoring:\n- Validates hyperparameter ranges against known constraints\n- Checks internal consistency (e.g., batch_size < buffer_size)\n- Uses LLM judge for reasoning quality assessment\n- Applies sigmoid smoothing for continuous scoring\n\n## Usage\n\n```python\nimport verifiers as vf\n\nenv = vf.load_environment(\"rl-hyperparameter-tuner\")\n# or\nfrom rl_hyperparameter_tuner import load_environment\nenv = load_environment()\n```\n\n## Version History\n\n- **v0.1**: Initial release with 6 algorithm families, 20+ scenarios, multi-dimensional scoring\n","encoding":"utf-8","truncated":false,"total_bytes":3212},"status":null}