{"data":{"kind":"file","path":"README.md","version_id":"a5pvhr79ed96qmg3c1nti50m","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4428,"modified_at":"2026-07-29T14:08:11.772000","content_hash":"1b8054b84a0dd386cc02cf4422af44c0ff8765fad12c3b388102185671ae4403"},"entries":[],"content":"# optimbench-dispatch\n\n### Overview\n- **Environment ID**: `optimbench-dispatch`\n- **Short description**: Dynamic vehicle dispatch as a verifiable, multi-turn RL environment: a capacitated routing problem with time windows and mid-episode disruptions, scored by a deterministic reward with no model-in-the-loop judge.\n- **Tags**: multi-turn, tool-use, verifiable, optimization, routing, dispatch, train, eval\n\n### Datasets\n- **Primary dataset(s)**: procedurally generated dispatch scenarios (no static corpus, so nothing to memorize or leak).\n- **Source links**: generated by the [`optimbench`](https://github.com/yferc/optimbench) library; each row is one seeded scenario.\n- **Split sizes**: evaluation uses the held-out `TEST_SEEDS` (50 per difficulty); training draws disjoint `TRAIN_SEEDS`. Difficulty is `easy`, `medium`, or `hard`.\n\n### Task\n- **Type**: multi-turn tool use. The agent assigns orders, sequences routes, and commits a plan through a ten-tool API, then must recover after each mid-episode disruption (a breakdown of the busiest vehicle, a rush order, or a cancellation).\n- **Output format expectations**: one JSON tool call per turn, e.g. `{\"action\": \"assign_order\", \"args\": {\"order_id\": \"ord_3\", \"vehicle_id\": \"veh_1\"}}`. Malformed replies are rejected, not crashed on.\n- **Rubric overview**: one scalar reward in `[0, 1]` = `integrity_gate * (0.7 * task + 0.3 * robustness)`. Integrity is a hard multiplicative gate (never committing, leaving a disruption unresolved, or spamming invalid actions scores zero). `task` is the committed cost against an agent-independent reference solve (capped at 1, zero if infeasible); `robustness` is the fraction of post-disruption waves left feasible. The three component scores are also logged as zero-weight metrics.\n\n### Quickstart\nRun an evaluation with default settings:\n\n```bash\nprime env install optimbench-dispatch\nprime eval run optimbench-dispatch -m openai/gpt-5-nano\nprime eval view\n```\n\n`load_environment(difficulty=\"medium\", max_turns_per_wave=80, train_size=128)` selects the tier. No API keys or secrets are required: the reward is fully deterministic and local.\n\n### Environment args\n| Field | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `difficulty` | str | `\"medium\"` | Scenario tier: `easy`, `medium`, or `hard`. |\n| `max_turns_per_wave` | int | `80` | Tool-call budget per wave before truncation. |\n| `train_size` | int | `128` | Number of `TRAIN_SEEDS` scenarios in the train split. |\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | Scalar in [0, 1]: integrity gate times weighted task and robustness. |\n| `task` | Committed cost ratio to the reference solve (1.0 = matches reference, 0 if infeasible). |\n| `robustness` | Fraction of post-disruption waves left feasible. |\n| `integrity` | 1.0 unless the run never committed, left a disruption unresolved, or spammed invalid actions. |\n\n### Baselines\nZero-shot reward on `medium`, test seeds 0-2. The hand-written and trained agents are scored by the\nsame reward function on the same seeds, so every row is comparable (reproduce the non-LLM rows with\n`python scripts/hub_baseline.py --seeds 3 --difficulty medium`).\n\n| agent | kind | reward |\n| ----- | ---- | ------ |\n| claude-opus-5 | LLM, zero-shot | **0.917** |\n| learned RL policy | trained | 0.870 |\n| deepseek-v3.2 | LLM, zero-shot | 0.766 |\n| greedy heuristic | hand-written | 0.762 |\n| gemini-2.5-flash-lite | LLM, zero-shot | 0.547 |\n| gpt-4.1-mini | LLM, zero-shot | 0.273 |\n| gpt-5-nano | LLM, zero-shot | 0.000 |\n| random | baseline | 0.000 |\n\nThree seeds is a pilot, not a stable estimate; LLM rows use provider-default sampling. The\nreproducible protocol is this environment's default eval config (20 examples x 3 rollouts) at a\nfixed temperature.\n\n### Notes\n- Reward is verifiable and reward-hacking-resistant by design (see the integrity axis above); there is no LLM judge.\n- One task separates models across the full range: a frontier model clears both the hand-written heuristic and the trained policy zero-shot, mid-tier models land around greedy, and weaker models collapse to zero. They collapse on *recovery*, not arithmetic: the plan is fine until the breakdown, then an order is left unassigned or a reloaded vehicle's route goes stale and the agent dispatches anyway. See the [OptimBench repo](https://github.com/yferc/optimbench) for the full leaderboard and design writeup.\n","encoding":"utf-8","truncated":false,"total_bytes":4428},"status":null}