{"data":{"kind":"file","path":"README.md","version_id":"fb8rnti3spknavrkurczj6ew","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3145,"modified_at":"2026-02-11T21:00:28.524000","content_hash":"e2265294838ebf40ac085fcec72d945943f1cb3e9230041eb6a26d38daf9c27d"},"entries":[],"content":"# homebench\n\n**Environment ID:** `homebench`  \n**Short description:** Multi-turn smart-home command generation benchmark with tool use and command-level scoring.  \n**Tags:** smart-home, tool-use, multi-turn, api-generation\n\n---\n\n## Overview\n\n- **Task type:** agentic tool-use command generation\n- **Primary data:** bundled HomeBench splits (`train`, `valid`, `test`)\n- **Interaction model:** query home state with tools, then finish with `execute_commands(commands)`\n- **Scoring model:** exact command match, set-level F1, and validity rates for rooms/devices/pairs\n\nThe environment is self-contained; datasets are included under `data/`.\n\n---\n\n## Quick Start\n\nRun an eval with defaults:\n\n```bash\nprime eval run homebench --model gpt-4.1-mini\n```\n\nWith environment args:\n\n```bash\nprime eval run homebench \\\n  --model gpt-4.1-mini \\\n  --split test \\\n  --max-samples 100 \\\n  --max-turns 10\n```\n\nInstall locally from path:\n\n```bash\nprime env install homebench -p ./environments/homebench\n```\n\n---\n\n## Environment Args\n\n| Arg | Type | Default | Description |\n|---|---|---:|---|\n| `split` | `str` | `test` | Dataset split: `train`, `valid`, `test` |\n| `max_samples` | `int` | `-1` | Max number of samples (`-1` = all) |\n| `max_turns` | `int` | `10` | Maximum turns before forced stop |\n| `system_prompt` | `str \\| null` | `null` | Optional prompt override |\n\n---\n\n## Tools\n\n- `list_rooms()`\n- `list_devices(room_name)`\n- `get_all_devices()`\n- `get_room_status(room_name)`\n- `get_device_status(room_name, device_name)`\n- `get_available_methods(room_name, device_name)`\n- `execute_commands(commands)` (terminal tool; ends episode)\n\n`commands` supports:\n- structured objects such as `{\"room_name\":\"balcony\",\"device_name\":\"light\",\"api_name\":\"set_brightness\",\"value\":40}`\n- legacy command strings such as `\"balcony.light.set_brightness(40)\"`\n- invalid-operation marker `\"error_input\"`\n\n---\n\n## Reward\n\nWeighted reward per example, max total **2.3**:\n\n| Metric | Weight | Range | Notes |\n|---|---:|---:|---|\n| `exact_match_reward` | 1.0 | 0..1 | Multiset exact match on predicted vs target commands |\n| `f1_reward` | 0.7 | 0..1 | Command-level F1 |\n| `valid_room_rate_reward` | 0.2 | 0..1 | Fraction of predicted commands with valid rooms |\n| `valid_device_rate_reward` | 0.2 | 0..1 | Fraction of predicted commands with valid devices |\n| `valid_pair_rate_reward` | 0.2 | 0..1 | Fraction of valid room-device pairs |\n\n---\n\n## Dataset Layout\n\n```text\ndata/\n├── home_status_method.jsonl\n├── test_data.jsonl\n├── valid_data.jsonl\n├── train_data_part1.jsonl\n└── train_data_part2.jsonl\n```\n\n---\n\n## Task I/O\n\nInput example:\n\n```text\nSet the brightness of the light to 20 in the bathroom.\n```\n\nOutput command examples:\n\n```text\n{bathroom.light.set_brightness(20)}\n```\n\n```text\n{error_input}\n```\n\nGlobal devices without room prefix are mapped to `room_name = \"None\"` internally (for example: `vacuum_robot.<api>(...)`).\n\n---\n\n## Publishing Notes\n\n- Build artifacts are constrained by `pyproject.toml` includes; `outputs/` is excluded from package build.\n- Local eval artifacts are ignored via `.gitignore` in this environment folder.\n","encoding":"utf-8","truncated":false,"total_bytes":3145},"status":null}