{"data":{"kind":"file","path":"README.md","version_id":"j54q2jb3sga04iagm571orrm","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6333,"modified_at":"2026-08-06T22:09:43.535000","content_hash":"ab746e0b987f28f9f77ff197f826159becd33768ef65108a2a896005108e4f52"},"entries":[],"content":"# Knowledge-Worker Search Bench\n\nThis benchmark evaluates the ability of LLMs to retrieve and synthesize knowledge across a wide variety of tool surfaces. All data here is purely synthetic, generated by [Tonic Fabricate](https://www.tonic.ai/fabricate) to showcase its ability to build longitudinally coherent narratives instantiated over multiple tool surfaces. \n\nThe data supporting these environments was built using a custom harness to drive Fabricate. Starting from a description of a persona, a narrative arc is generated in increasingly granular detail. The events in this story drive the generation of tool artifacts, such as email, slack, calendar events, etc. The result is a SQLite database of coherent artifacts with associated narrative metadata. This combination allows us to scale the generation of search tasks by synthesizing user questions based on narrative ground truth and verifying that the ssociated artifacts faithfully represent and support the facts that underly a correct answer. Answers are graded on whether or not they contain the relevant factual claims and are correctly supported by relevant artifact citations. Agents completing the task must use MCPs and other tool surfaces to interact with artifacts, and never see the underlying narrative metadata. \n\nEach task drops an agent into one persona's work world — mail (Outlook or\nGmail), Slack, Google Docs, calendar, attachments — and asks a question a real\nchief-of-staff-style assistant would get: *\"brief me for tomorrow's sync\"*,\n*\"where did we land on the renewal, and what forced the timeline?\"*. Answering\nrequires finding and synthesizing evidence scattered across channels; grading\nis per-claim (an LLM judge checks each rubric claim is conveyed and grounded\nin cited artifacts).\n\n## Where things live\n\n| surface | contents |\n|---|---|\n| **This repo** (central source) | the eval environment (a [verifiers](https://github.com/PrimeIntellect-ai/verifiers) env: tools, per-claim rubric judge, task loader), runner, docs, results, task viewer |\n| **HF dataset** [`TonicAI/knowledge-worker-search-bench`](https://huggingface.co/datasets/TonicAI/knowledge-worker-search-bench) | the tasks (browsable `tasks.jsonl` + runtime specs) and the 10 persona SQLite environments |\n| **Environments Hub** [`tonic-ai/knowledge-worker-search-bench`](https://app.primeintellect.ai/dashboard/environments/tonic-ai/knowledge-worker-search-bench) | the installable environment package (`prime env install tonic-ai/knowledge-worker-search-bench`) |\n\nThe harness resolves tasks and persona DBs from the HF dataset automatically\n(`huggingface_hub`, cached locally). Overrides: `MULTICHANNEL_SEARCH_DATA_DIR`\n(local tasks dir), `MULTICHANNEL_SEARCH_PERSONAS_DIR` (local DBs),\n`MULTICHANNEL_SEARCH_HF_REPO` (different dataset repo).\n\n```\nmultichannel_search/            the eval environment package\nscripts/run_eval.py             run models over the tasks\nresults/                        per-task rewards for 6 models + selection provenance\nviewer/                         self-contained task viewer (index.html + viewer_data.json)\n```\n\n## Run the tasks\n\n```bash\n# from the Environments Hub\nprime env install tonic-ai/knowledge-worker-search-bench\n# or from this repo\npip install -e .          # or: uv pip install -e .\n\nexport OPENAI_API_KEY=... # judge + OpenAI models\nexport ANTHROPIC_API_KEY=... # for claude-* models\nhuggingface-cli login     # while the dataset is private\n\npython scripts/run_eval.py \\\n  --models claude-haiku-4-5 \\\n  --tasks becca-dipti-cross-functional-sync-brief \\\n  --concurrency 4\n\n# or with the verifiers / Prime tooling:\nvf-eval knowledge_worker_search_bench --provider anthropic --model claude-haiku-4-5\nprime eval run knowledge_worker_search_bench --provider anthropic --model claude-haiku-4-5\n```\n\nOmit `--tasks` to run the full 40-task slate. The env is loaded via\n`multichannel_search.load_environment()`, so it also plugs into any\nverifiers-compatible harness (including RL trainers). The agent gets read-only\ntools over the persona DB (mail/Slack/docs/calendar search + read, attachment\nparsers) and must finish with `workspace__write_output`; the rubric judge\n(gpt-4.1-mini, temp 0) then grades each claim.\n\n## Browse the tasks\n\n```bash\nbash viewer/serve.sh          # -> http://127.0.0.1:8899\n```\n\nThe viewer shows, per task: the instruction, the rubric claims, every evidence\nartifact with its full body (color-coded by channel), and per-model rollouts —\nfull multi-turn tool-use traces where captured, with per-claim grade tables.\nFor a quick look without the viewer, the HF dataset's `tasks.jsonl` renders in\nthe HF dataset viewer.\n\n## Model results (this 40-task slate)\n\nRewards from the unified per-claim conveyed/grounded grader; single rollout per\ntask, verifiers harness. `pass@1` = fraction of tasks with reward 1.0.\n\n| model | mean reward | pass@1 |\n|---|--:|--:|\n| gpt-5.5 | 0.92 | 62.5% |\n| claude-opus-4.7 | 0.89 | 70.0% |\n| claude-sonnet-4.6 | 0.70 | 30.0% |\n| claude-haiku-4.5 | 0.56 | 20.0% |\n| gpt-5.4-mini | 0.46 | 12.5% |\n| Qwen3.6-35B-A3B | 0.39 | 17.5% |\n\nEvery task is verified solvable — at least one frontier model scores ≥ 0.8 —\nyet the slate separates models cleanly.\n\n## Task anatomy\n\nA task spec (`tasks/<task_id>.json` in the HF dataset) contains:\n\n- `system` — the persona framing (who the agent works for, what surfaces exist)\n- `user` — the request, plus the required `workspace__write_output` JSON schema\n- `ground_truth.rubric_claims` (or `expected_claims` / `gold_answer` by shape) —\n  what a correct answer must convey\n- `ground_truth.source_artifact_ids` — the evidence artifacts, resolvable in\n  the persona DB\n\nShapes: `multi_hop_chain` (32) — evidence chained across channels;\n`thread_synthesis` (6) — synthesize one conversation thread;\n`attachment_lookup` (2) — find a fact inside an email attachment.\n\nDifficulty bands (easy 20 / medium 18 / hard 2) are empirical: mean reward\nacross all six evaluated models on the parent corpus.\n\n## The environments\n\nPersona DBs are fully synthetic worlds: consistent orgs, characters,\nstorylines and timelines woven across `outlook__*` / `gmail__*` / `slack__*` /\n`google_docs__*` / `*calendar__*` tables. No real people, companies, or data.\nAgents reach them only through read-only product-shaped tools, never raw SQL.\n","encoding":"utf-8","truncated":false,"total_bytes":6333},"status":null}