{"data":{"kind":"file","path":"README.md","version_id":"aa6xxsne1p6fhrq5scr0s8ts","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5668,"modified_at":"2026-08-07T18:11:02.745000","content_hash":"73fd3406ee631ea0f5ff5144df8458d5810fac2ce6a9fb30bce095ace3db04c5"},"entries":[],"content":"# service-desk-bench\n\n### Contributed by\n- **Sarvesh Patel** — GitHub: https://github.com/swami1234\n\nAn agent runs the **front office of a small home-services business**. Each episode\nseeds a lead inbox for a real-world trade (towing, junk removal, dumpster rental,\ngutter cleaning) with a mix of quotable jobs, jobs missing required information,\nout-of-service-area requests, and spam. The agent must triage every lead, ask\ncustomers for missing details **before** quoting, price quotes **exactly** from the\nbusiness's pricing playbook, and finish with a report.\n\nCustomer replies are **scripted and deterministic** (keyword-triggered reveal rules),\nso episodes are fully reproducible and require no second agent model. Task realism is\ndrawn from operating real contractor lead/quote workflows in production; all data is\nsynthetic (no real PII).\n\n### Why this task\nMost environments on the Hub test coding, math, games, or general web navigation.\nAlmost none test the messy SMB operations work — triage, clarification, exact\nplaybook pricing, judgment about spam and serviceability — where much of the\nnear-term economic value of agents sits. The failure modes it surfaces are distinct:\nquoting without required info, inventing prices not in the playbook, misclassifying\nedge cases (heavy-debris rules, after-hours surcharges), and re-asking for details\nthe customer already gave.\n\n### Task\n- **Type**: multi-turn tool use (`StatefulToolEnv`), no sandbox required\n- **Tools**:\n  - `read_inbox()` — list all leads with id, channel, text, status\n  - `get_pricing_playbook()` — business profile, service-area ZIPs, prices, rules\n  - `reply_customer(lead_id, message)` — message a customer; scripted reply may reveal missing info\n  - `send_quote(lead_id, line_items, total)` — submit a structured quote (JSON line items + total)\n  - `set_status(lead_id, status)` — NEW / NEED_INFO / QUOTED / DECLINED / SPAM\n  - `final_report(summary)` — end the episode\n- **Episode ends** when `final_report` is called or `max_turns` is reached.\n\n### Dataset\n`scenarios.jsonl` — 4 scenarios / 22 leads (v0.1). Each lead carries ground truth:\nfinal status, exact quote total (from the playbook), required clarifying asks with\nkeyword triggers + scripted reveals, and provided-info keywords for redundancy\ndetection. The schema doubles as the grader specification.\n\n### Rubric\nDeterministic-heavy by design (structured comparisons, hard to reward-hack):\n\n| Reward | Weight | How |\n| --- | --- | --- |\n| Quote math correctness | 0.45 | submitted `total` vs ground-truth total (±$0.01), per quotable lead |\n| Triage correctness | 0.25 | final status vs ground-truth status, per lead |\n| Info-gathering discipline | 0.15 | required asks made before quoting; penalty for re-asking provided info |\n| Communication quality | 0.15 | LLM judge (optional — see below), structured verdict good/acceptable/poor |\n\nPlus a 0-weight `leads_handled` observability metric. **If no judge key is set, the\nenvironment runs fully deterministic** and the three deterministic weights are\nrenormalized — contributors can run it with zero external credentials.\n\n### Quickstart\n```bash\nuv run vf-eval -s service-desk-bench -m gpt-4.1-mini -n 4 -r 2\n```\n\nWith the communication judge enabled:\n```bash\nexport JUDGE_API_KEY=...   # Gemini key by default (OpenAI-compatible endpoint)\nuv run vf-eval -s service-desk-bench -m gpt-4.1-mini -n 4 -r 2\n```\n\nRun a subset of scenarios:\n```bash\nuv run vf-eval -s service-desk-bench -a '{\"scenario_ids\": [\"towing-raleigh-01\"]}'\n```\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `dataset_path` | str or null | bundled `scenarios.jsonl` | Override scenarios file |\n| `scenario_ids` | list[str] or null | all | Subset of scenarios to run |\n| `max_turns` | int | `30` | Max assistant turns per episode |\n| `system_prompt` | str or null | built-in | Override operator instructions |\n| `use_judge` | bool or null | auto | Force communication judge on/off (auto = on iff key present) |\n| `judge_model` | str | `gemini-2.5-flash-lite` | Judge model |\n| `judge_api_key_var` | str | `JUDGE_API_KEY` | Env var holding judge API key |\n| `judge_base_url` | str or null | Gemini OpenAI-compat URL | Judge endpoint |\n\n### Baseline results (v0.1, 4 scenarios × 1 rollout, deterministic rubric, no judge)\n| Model | Overall | Quote math | Triage | Discipline |\n| --- | --- | --- | --- | --- |\n| gpt-4.1-mini | **0.81** | 0.71 | 0.95 | 0.90 |\n| gpt-4.1-nano | **0.27** | 0.06 | 0.28 | 0.86 |\n\nThe 3× spread shows the benchmark discriminates by capability: the weak model rarely\nsubmits structured quotes at all (0.5 `send_quote` calls/episode vs 2.75) and\nmis-triages most leads, while the strong model's remaining errors are genuine playbook\nmistakes (missed surcharges, wrong load size) — the exact behaviors the env targets.\n\n### Grading notes / anti-reward-hacking\n- Quote totals are compared as structured numbers from `send_quote`, never parsed from prose.\n- Ground truths are precomputed in the dataset; the grader does no arithmetic of its own.\n- Spam customers never reply; declined/spam leads have `quote_total: null` so quoting them\n  earns nothing.\n- Redundancy detection uses question-phrased keywords (e.g. \"how many miles\") to avoid\n  penalizing legitimate quote messages that mention provided facts.\n\n### Roadmap\n- v0.2: 12 scenarios / ~70 leads across 8 trades; ambiguity/difficulty knobs\n  (`spam_rate`, `ambiguity_level`); harder playbooks (tiered pricing, minimums).\n- v0.3: adversarial customers (haggling, contradictory info), multi-lead\n  interleaving pressure, RL-training configs.\n","encoding":"utf-8","truncated":false,"total_bytes":5668},"status":null}