{"data":{"kind":"file","path":"README.md","version_id":"x6ybhvto09v2kq5o94j9aloc","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10160,"modified_at":"2026-08-20T12:47:45.497000","content_hash":"d891d996073854c45bea93a5a11c9d405928ee07025c56de19e7c70dac461f87"},"entries":[],"content":"# itsm-bench\n\n### Overview\n\n- **Environment ID**: `itsm-bench`\n- **Short description**: A live ITSM tenant with 93 typed tools, a written policy, and a simulated operator — 53 multi-turn tasks graded on the final database state, not the transcript.\n- **Tags**: `multi-turn`, `tool-use`, `agent`, `enterprise`, `itsm`, `user-simulation`, `train`, `eval`\n\nMost agent benchmarks stop at the answer. Real organizations don't — work only counts once it has been carried out in a system of record, by someone allowed to do it, with a trail that survives an audit. ITSMBench measures that operational layer: the agent is dropped into a live IT Service Management tenant (~20 interconnected tables: incidents, SLA clocks, problems, change requests, a CMDB, a service catalog, knowledge articles, notifications, users, groups, roles, permissions), handed a written policy it is held to, and asked to carry out a request from a simulated operator who only volunteers what they are asked for. Nothing is scored from the conversation; every task is graded on the state the agent leaves behind.\n\nThis environment is the [Enterprise-Worlds](https://github.com/vibrantlabsai/Enterprise-Worlds) ITSMBench world, packaged for the Prime Intellect Environments Hub. A run through this env sees the identical policy text, tool schemas, user-sim prompts, and reward as the upstream `eworlds run` harness. See the [live leaderboard](https://enterpriseworlds.vibrantlabs.com/) for reference scores (`pass@k` / `pass^k` over four independent trials).\n\nThe ITSM seed databases and typed tool surface are adapted from ServiceNow's [EnterpriseOps-Gym](https://github.com/ServiceNow/EnterpriseOps-Gym) (Apache-2.0), reimplemented as in-memory Python tool calls — no Docker or SQL server at runtime. The multi-turn user-simulator loop, org-scoped multi-tenant task model, and combined DB-state + natural-language-assertion verifier are original to Enterprise-Worlds, in the style of [τ-bench](https://arxiv.org/abs/2406.12045).\n\n### Datasets\n\n- **Primary dataset**: the ITSMBench benchmark set — **53 tasks**, bundled inside the package (no downloads, no external data dependencies).\n  - **31 MSP tasks** (`itsmbench_msp_*`): a managed-service-provider world serving 20 client organizations, seeded from `msp_db.json`. Tasks cross tenant boundaries; access scope is part of the difficulty.\n  - **22 single-tenant tasks** (`itsmbench_single_*`): a single-company world seeded from `single_tenant_db.json`.\n- **Source links**: [Enterprise-Worlds repo](https://github.com/vibrantlabsai/Enterprise-Worlds) · [leaderboard](https://enterpriseworlds.vibrantlabs.com/)\n- **Row structure**: each dataset row is a system prompt (the agent instruction with the full ITSM policy embedded) plus the serialized task — operator persona and private `known_info`, seed world, tenancy scope (`org_ids`), frozen clock (`current_time`), and evaluation criteria. Each task carries its own seed and clock, so single-tenant and cross-org tasks run side by side and every timestamp a run stamps is reproducible.\n\n### Task\n\n- **Type**: multi-turn tool use with a simulated user (native tool calling — the agent model must support it)\n- **Tools**: 93 typed tools for lookup, mutation, and aggregation, shared across all tasks. They enforce enum gates, required fields, and referential integrity like the real system would, and generate IDs and timestamps deterministically so runs stay comparable.\n- **Episode flow**:\n  1. The rollout opens with the canned greeting `\"Hi! How can I help you today?\"` followed by the simulated operator's first message.\n  2. Each agent turn either issues tool calls (executed against the live in-memory tenant; errors come back as `\"Error: ...\"` tool messages, never exceptions) or sends text, which is answered by the user simulator. The simulator has a persona and private `known_info`, disclosed progressively — facts surface only when the agent asks the right question.\n  3. The episode ends when the agent or the simulated user emits `###STOP###`, or when `max_turns` assistant turns are reached (truncated episodes are still scored).\n- **User simulator**: any OpenAI-compatible chat model (`user_model`, default `gpt-4o-mini`, temperature 0.7). Requires an API key in the environment (`OPENAI_API_KEY` by default; configurable via `user_api_key_var` / `user_base_url`).\n\n### Rubric\n\nReward is **strictly binary**: the product of every criterion the task defines — all must pass.\n\n- **DB match** (`db_match`): the task's gold action sequence is replayed on a fresh seed to produce the expected final database, which is compared to the run's final database. Structured fields must match exactly; free-text fields are graded by a semantic LLM judge (`db_text_match=\"llm\"`, the default), so reworded-but-correct prose still passes. Set `db_text_match=\"fuzzy\"` for judge-free, fully deterministic scoring, or `\"exact\"` for strict string equality.\n- **NL assertions** (`nl_assertions`): an LLM judge grades each of the task's natural-language assertions against the conversation transcript — covering what the database can't express. Set `skip_nl_assertions=true` for a DB-only reward (upstream's gym mode). Note it takes **both** flags to remove the judge entirely: with the default `db_text_match=\"llm\"`, free-text database fields are still judged.\n\nThe judge (`judge_model`, default `gpt-4o-mini`) runs at temperature 0. If no judge is usable while `db_text_match=\"llm\"`, text matching degrades to fuzzy rather than crashing.\n\n### Quickstart\n\nInstall from the Hub:\n\n```bash\nprime env install vibrantlabsai/itsm-bench\n```\n\nAn OpenAI-compatible key is required for the user simulator and the judge. Export it **before** loading the environment — judge credentials are resolved when `load_environment` runs (loading without a key still works, but the judge stays disabled for that run: DB free-text matching degrades to fuzzy, and tasks with NL assertions score 0 unless `skip_nl_assertions` is set):\n\n```bash\nexport OPENAI_API_KEY=sk-...\n```\n\nRun an evaluation with the default settings (temperature 0 is recommended for the agent):\n\n```bash\nvf-eval itsm-bench -m gpt-4o -n 5 -r 2 -T 0 -a '{\"user_model\": \"gpt-4o-mini\"}'\n```\n\nDeterministic, judge-free scoring (DB-only reward, no LLM judge calls):\n\n```bash\nvf-eval itsm-bench -m gpt-4o -n 5 -r 2 -T 0 \\\n  -a '{\"db_text_match\": \"fuzzy\", \"skip_nl_assertions\": true}'\n```\n\nEach task carries its own seed world, so the task set is subset-selectable but the seed is not overridable. Restrict to specific tasks:\n\n```bash\nvf-eval itsm-bench -m gpt-4o -T 0 -a '{\"task_ids\": [\"itsmbench_msp_001\", \"itsmbench_single_001\"]}'\n```\n\nNotes:\n- Use `-a` / `--env-args` to pass environment arguments as a JSON object.\n- The agent model must support native tool calling.\n\n### Environment Arguments\n\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `task_ids` | list[str] \\| None | `None` | Restrict to a subset of task IDs (e.g. `\"itsmbench_msp_001\"`) |\n| `max_turns` | int | `12` | Maximum assistant turns per episode (tool rounds + text replies, ≈ upstream `--max-steps 12`); truncated episodes are still scored |\n| `user_model` | str | `\"gpt-4o-mini\"` | User-simulator model (OpenAI-compatible) |\n| `user_base_url` | str \\| None | `None` | Base URL for the user-sim client (`None` = OpenAI default) |\n| `user_api_key_var` | str | `\"OPENAI_API_KEY\"` | Env var holding the user-sim API key |\n| `user_temperature` | float | `0.7` | User-simulator sampling temperature (upstream default) |\n| `user_sampling_args` | dict \\| None | `None` | Override user-sim decode params; `{\"temperature\": null}` omits one for models that reject it |\n| `judge_model` | str | `\"gpt-4o-mini\"` | Judge model for DB free-text grading and NL assertions |\n| `judge_base_url` | str \\| None | `None` | Base URL for the judge client (`None` = OpenAI default) |\n| `judge_api_key_var` | str | `\"OPENAI_API_KEY\"` | Env var holding the judge API key (no fallback: if it is unset, the judge is disabled rather than reaching for `OPENAI_API_KEY`) |\n| `judge_sampling_args` | dict \\| None | `None` | Override judge decode params; `{\"temperature\": null}` omits one |\n| `db_text_match` | str | `\"llm\"` | Free-text field comparison in the DB check: `\"llm\"` (semantic judge), `\"fuzzy\"` (deterministic, judge-free), or `\"exact\"` |\n| `skip_nl_assertions` | bool | `false` | `true` skips the NL-assertion judge — DB-only reward. Pair with `db_text_match=\"fuzzy\"` for a fully judge-free run |\n\nAdditional keyword arguments are forwarded to the underlying `vf.Environment`.\n\n### Metrics\n\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Binary task success: `db_match × nl_assertions` over the criteria the task defines — 1.0 only if every defined criterion passes |\n| `db_match` | 1.0 if the final database matches the gold-replay state (0-weight diagnostic; 1.0 when the task defines no DB criterion) |\n| `nl_assertions` | 1.0 if every NL assertion passes the judge (0-weight diagnostic; 1.0 when absent or skipped) |\n| `scoring_ok` | 1.0 when the evaluator ran cleanly; 0.0 when scoring itself failed (judge outage, missing credentials). Watch this before reading a run's `reward` — a broken harness and a failing model both report 0 otherwise |\n\nThe evaluation is unforgiving by design — partial success scores 0.\n\n### Credits\n\nBuilt by Vibrant Labs as part of [Enterprise-Worlds](https://github.com/vibrantlabsai/Enterprise-Worlds). Seed databases and tool surface adapted from ServiceNow's [EnterpriseOps-Gym](https://github.com/ServiceNow/EnterpriseOps-Gym) ([arXiv:2603.13594](https://arxiv.org/abs/2603.13594), Apache-2.0); verifier design in the style of [τ-bench](https://arxiv.org/abs/2406.12045). Licensed Apache-2.0.\n\n```bibtex\n@misc{enterpriseworlds2026,\n      title={{Enterprise-Worlds: Executable Enterprise Environments for Measuring Operational Agents}},\n      author={Shahul Elavakkattil and Ankit Sridhar and Andrew Bastin and Jithin James and Kumar Anirudha and Arjun Devarajan},\n      year={2026},\n      publisher={Vibrant Labs},\n      url={https://github.com/vibrantlabsai/Enterprise-Worlds},\n}\n```\n","encoding":"utf-8","truncated":false,"total_bytes":10160},"status":null}