{"data":{"kind":"file","path":"README.md","version_id":"t4zdbq6l9glg8zk6bfb7r51n","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2165,"modified_at":"2026-09-13T04:06:06.966000","content_hash":"31837852c7f38b7a1c35bc7b1e55d3f71053617eaf4558eee09bbcafcad6ba5d"},"entries":[],"content":"# tau2-synth\n\ntau2-bench with custom synthetic domains as a native Verifiers v1 taskset and harness, with multi-turn dual-control tool use driven by an LLM user simulator. Tasks are scored with the official tau2 evaluation of database state, environment assertions, actions, and required communication.\n\n## Taskset\n\n- **Source:** [`mikasenghaas/tau2-synth`](https://github.com/mikasenghaas/tau2-synth) (pinned `base` split per domain)\n- **Size:** Tasks across the 6 synthetic domains defined in the taskset (`library`, `fitness_gym`, `tech_support`, `cloud_incident_response`, `daily_planner`, `ev_charging_support`)\n\n## Harness\n\nShips a custom `Tau2SynthHarness` (`vf.Harness`) that runs tau2's official simulation over the synthetic domains: the evaluated agent is routed through the verifiers endpoint while an LLM user simulator (default `gpt-4.1`) drives dual-control multi-turn tool use against the domain database. The full simulation and evaluation breakdown is stored in `trace.info[\"tau2\"]`.\n\n## Notes\n\n- Domain data is downloaded once at runtime from the pinned tau2-synth revision (branch `synth`) into `~/.cache/tau2-synth/data`, guarded by a file lock and revision marker so concurrent rollouts bootstrap it exactly once.\n- Tau simulations allow up to 500 orchestrator steps (raised from tau's default 100) so long scenarios can finish while non-terminating conversations stay bounded; this counts all agent, user, and tool transitions, not just model turns.\n- Without Prime credentials the user simulator falls back to `OPENAI_API_KEY` (and optional `OPENAI_BASE_URL`).\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-08-27: Pin the environment's agent runtime to a subprocess so the bundled Tau orchestrator uses the interpreter that installed it.\n- 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the `Task` (a `TaskData` row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from `--env.taskset.*` to `--env.taskset.task.*`. Requires `verifiers>=0.2.0` and Python `>=3.11`.\n","encoding":"utf-8","truncated":false,"total_bytes":2165},"status":null}