{"data":{"kind":"file","path":"README.md","version_id":"prfgo9azwnm9d58mhmsnosjp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4022,"modified_at":"2026-07-23T20:46:16.787000","content_hash":"6f8b19a631b08443195e4f3266ff8590f5bf626646c16e0ff63ecd7479633166"},"entries":[],"content":"# TauLong Bench v1\n\nTauLong Bench evaluates whether an agent can retain and selectively reuse\nexperience across sequential customer-support cases. TauLong owns the stream,\nmemory boundary, and one masked-evidence rule. The agent loop, user simulator,\ntelecom tools, and evaluator are from the pinned tau2 revision. No LLM judge\ncontributes to reward.\n\n## Task design\n\nOne task is a shift of eight customers. Every customer starts with a fresh\nconversation, environment, and user state. Raw messages from earlier customers\nare discarded.\n\nThe default shift contains three populations:\n\n- **Teaching incidents** expose the recurring cause through normal diagnostics.\n- **Testing incidents** withhold that diagnostic evidence, so retained\n  experience is required to improve beyond one-of-four guessing.\n- **Lookalikes** present a similar symptom but require a different fix.\n\nThe only cross-case state in the default mode is a model-written notebook capped\nat 2,000 characters.\n\nUser personas are normalized across causes so customer behavior cannot reveal a\nhidden diagnosis. In testing incidents, diagnostic and verification results are\nunavailable until the customer commits to one targeted setting change. Routine\nreset steps still execute, but expose no device status.\n\nIncident causes are balanced in independently shuffled blocks. Fixture identities\ncome from a separate stable hash, so customer identifiers expose no arithmetic\nencoding of the shift seed or recurring cause.\n\nEach case allows nine agent utterances addressed to the customer after the\nopening greeting. Phone-tool execution, backend lookups, and notebook writes do\nnot consume customer patience. A separate 40-step tau2 orchestration limit bounds\ninternal work.\n\n## Scoring\n\nThe training reward is the sum of official tau2 `EnvironmentEvaluator` scores\nover all cases in the shift. Notebook use and intermediate behavior receive no\ndirect reward.\n\nThe task also reports:\n\n| Metric | Definition |\n| --- | --- |\n| `masked_env_accuracy` | Verified resolution rate on testing incidents |\n| `teaching_env_accuracy` | Verified resolution rate on teaching incidents |\n| `lookalike_env_accuracy` | Verified resolution rate on lookalikes |\n| `masked_transfer_rate` | Transfer rate on testing incidents |\n\n## Memory modes\n\n| Mode | Cross-case state |\n| --- | --- |\n| `none` | None |\n| `notebook` | Notes written by the evaluated model |\n| `oracle` | Read-only notebook initialized with the recurring fix |\n\nUse the three modes on matched seeds when measuring the value of model-written\nmemory.\n\nThe taskset exposes disjoint `train` and `eval` seed ranges. Training defaults to\n`--taskset.split train`; held-out evaluation uses `--taskset.split eval`. A\ncustom non-negative `--taskset.seed_offset` is available for additional disjoint\nsweeps.\n\nThe calibrated defaults live in `ShiftConfig` and the taskset exposes the case\ncount, customer-turn cap, internal-step limit, incident fraction, and teaching\ncount. Stream placement, fixture identities, oracle procedures, and\ncorrective-tool classes come from configuration or the pinned tau2 task and tool\nmetadata.\n\n## Run the published environment\n\n```bash\nprime env pull dennwslee/taulong_bench_v1@0.3.1 --target taulong-bench-v1\ncd taulong-bench-v1\nuv sync\n\nuv run eval taulong-bench-v1 \\\n  -m qwen/qwen3-30b-a3b-instruct-2507 \\\n  -n 1 -r 1 --rich false \\\n  --taskset.num_shifts 1 \\\n  --taskset.split eval \\\n  --taskset.memory_mode notebook\n```\n\nRun matched ablations by changing `--taskset.memory_mode` to `none`,\n`notebook`, or `oracle`.\n\n## Develop and publish\n\nFrom this repository:\n\n```bash\ncd environments/taulong_bench_v1\n./prepare_release.sh\ncd dist/release\nuv run --group dev pytest -q\n```\n\n`prepare_release.sh` creates the standalone artifact under `dist/release` with\nthe environment and telecom fixtures. `push.sh` tests, builds, and publishes\nthat same directory.\n\n## Compatibility\n\n- Python 3.12 or 3.13 is required.\n- The evaluated agent and customer simulator require configured model endpoints.\n","encoding":"utf-8","truncated":false,"total_bytes":4022},"status":null}