{"data":{"kind":"file","path":"README.md","version_id":"u9e0dlhpn9836h3t8z0l9f29","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3475,"modified_at":"2026-09-16T12:25:28.071000","content_hash":"e38228c01658604b8201f8a2eee35e33e9e30893f94b727dd4e2aba709e06738"},"entries":[],"content":"# browsecomp-plus\n\nBrowseComp-Plus ([arXiv:2508.06600](https://arxiv.org/abs/2508.06600), [texttron/BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus)) deep-research QA: 830 BrowseComp queries re-grounded in a fixed, human-verified corpus of 100,195 web documents. Unlike `browsecomp` (bring-your-own web search), the taskset serves the benchmark's own retrieval tool — a shared BM25 `search` tool over the corpus (top-5 hits with 512-token snippets; optional `get_document`) — so runs are reproducible and the retriever is a controlled variable. The agent answers in chat in BrowseComp's `Explanation / Exact Answer / Confidence` format (citing evidence docids) and the last reply is graded by the benchmark's HLE-style LLM judge against the decrypted gold answer. Metrics mirror upstream: binary `correct` (accuracy), `recall` (human-verified evidence docs surfaced by `search`), `num_search_calls`, and `model_confidence`.\n\n## Taskset\n\n- **Source:** [Tevatron/browsecomp-plus](https://huggingface.co/datasets/Tevatron/browsecomp-plus) (XOR-obfuscated, decrypted at load) over [Tevatron/browsecomp-plus-corpus](https://huggingface.co/datasets/Tevatron/browsecomp-plus-corpus) (100,195 docs, ~1.8 GB)\n- **Size:** 830 tasks\n- **Setup:** first load downloads the corpus and builds the BM25 index into `~/.cache/browsecomp-plus` (override with `BROWSECOMP_PLUS_CACHE_DIR`) — a one-time cost of a few minutes\n- **Config:** `--env.taskset.tools.k` (hits per search, default 5), `--env.taskset.tools.snippet-max-tokens` (default 512), `--env.taskset.tools.include-get-document` (default false; also switches the prompt to the get_document variant), `--env.taskset.task.judge.*` (grader).\n- **Harness:** the taskset bundles the null harness (a plain tool-calling chat loop, matching the upstream reference agent), so bare `uv run eval` runs the benchmark's intended setting.\n\n## Benchmark-faithful eval\n\n```bash\nuv run eval browsecomp-plus -m <model> --max-turns 100 --sampling.max-tokens 10000 \\\n  --env.taskset.task.judge.model openai/gpt-4.1\n```\n\nUpstream allows 100 agent iterations and 10k completion tokens per call, and grades with gpt-4.1 (the current leaderboard standardizes on Qwen3-32B; the authors report <1% judge disagreement). Reference anchor: gpt-4.1 + BM25 scores 14.58% accuracy / 16.42% evidence recall / 10.35 search calls; this port measured 17.23% / 18.12% / 9.94 under the same settings (see Deviations).\n\n## Deviations\n\n- Retrieval uses `bm25s` with Lucene-equivalent scoring (`method=\"lucene\"`, k1=0.9, b=0.4 — Pyserini's defaults) instead of upstream's Pyserini/Lucene (a Java dependency), so rankings can differ slightly from the official BM25 index.\n- The harness namespaces MCP tools by server, so the model sees `browsecomp_plus_search` (and `browsecomp_plus_get_document`) instead of upstream's bare `search`/`get_document`.\n\n## Changelog\n\n- 2026-08-31: Dropped the stale FastMCP comment left from the `mcp>=1,<2` pin era; verifiers main runs the tool server on `mcp.server.mcpserver.MCPServer` from mcp 2.\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-08-30: Standardized grading on `ReferenceJudge` while keeping the official HLE-style prompt and explicit `correct` reward in the taskset.\n- 2026-07-31: Migrated the shared search service to the explicit taskset-level `toolsets(config)` API required by `verifiers>=0.2.2.dev65`.\n- 2026-07-15: Initial v1 taskset.\n","encoding":"utf-8","truncated":false,"total_bytes":3475},"status":null}