{"data":{"kind":"file","path":"README.md","version_id":"cvnz88uk167agm6vs60t4m17","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":13924,"modified_at":"2026-06-26T18:02:34.694000","content_hash":"2c0a8972f9482d3be801ec9e3c7266c65865cdcc07e16236bd59dc0174f66370"},"entries":[],"content":"# web-search-env\n\n`web-search-env` is a Verifiers environment for search-based QA tasks. It gives\nthe model a single `search(query)` tool and keeps the search provider hidden\nbehind a provider-agnostic backend.\n\nUse it to evaluate the same agent with different search providers, or to run the\nbenchmark suite from Exa's search RL blog post.\n\n## Install\n\n```bash\nprime env install web-search-env\n```\n\n## Search Providers\n\nSet `search_backend` to one of:\n\n- `fixture`: local deterministic search, no API key required\n- `exa`: Exa search via `EXA_API_KEY`\n- `parallel`: Parallel search via `PARALLEL_API_KEY`\n- `serpapi`: Google results through SerpApi via `SERPAPI_API_KEY`\n- `serper`: Google results through Serper via `SERPERAPI_API_KEY`\n\nExample:\n\n```bash\nprime eval run web-search-env \\\n  -m openai/gpt-4.1-mini \\\n  -n 20 -r 1 -t 1024 \\\n  -a '{\"eval_dataset\":\"hotpotqa\",\"search_backend\":\"exa\",\"judge_backend\":\"llm\"}'\n```\n\nUse the same command with only `search_backend` changed to compare providers.\n`cache_path` can be set to reuse live search results across repeated runs.\n\n## Datasets\n\nSupported datasets:\n\n- `sample`\n- `simpleqa`\n- `2wikimultihopqa` or `2wiki`\n- `hotpotqa`\n- `frames`\n- `musique`\n- `browsecomp`\n- `hle`\n- `widesearch`\n\nBrowseComp is packaged with the environment, so it does not download the CSV at\nruntime.\n\nHLE uses the official `cais/hle` dataset. That dataset is gated on Hugging Face:\naccept access there and set `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` before\nrunning it. The current loader skips image rows because this environment does\nnot pass images to the model.\n\nWideSearch uses the official `ByteDance-Seed/WideSearch` dataset and downloads\nthe matching gold CSV tables from the same Hugging Face repo. Use an LLM judge\nfor this dataset; exact or normalized string matching is not meaningful for\nlarge table outputs.\n\nHLE and WideSearch are opt-in only. They are not part of the default sample\nrun, the Exa blog suite, or the Exa training setup.\n\nRun a single dataset:\n\n```bash\nprime eval run web-search-env \\\n  -m openai/gpt-4.1-mini \\\n  -n 50 -r 1 -t 1024 \\\n  -a '{\n    \"eval_dataset\":\"musique\",\n    \"search_backend\":\"parallel\",\n    \"parallel_mode\":\"advanced\",\n    \"judge_backend\":\"llm\"\n  }'\n```\n\nRun multiple datasets:\n\n```bash\nprime eval run web-search-env \\\n  -m openai/gpt-4.1-mini \\\n  -n 100 -r 1 -t 1024 \\\n  -a '{\"eval_dataset\":\"musique,hotpotqa\",\"search_backend\":\"exa\",\"judge_backend\":\"llm\"}'\n```\n\nRun HLE:\n\n```bash\nprime eval run web-search-env \\\n  -m openai/gpt-4.1-mini \\\n  -n 20 -r 1 -t 4096 \\\n  -a '{\"eval_dataset\":\"hle\",\"search_backend\":\"exa\",\"judge_backend\":\"llm\"}'\n```\n\nRun WideSearch:\n\n```bash\nprime eval run web-search-env \\\n  -m openai/gpt-4.1-mini \\\n  -n 5 -r 1 -t 8192 \\\n  -a '{\"eval_dataset\":\"widesearch\",\"search_backend\":\"exa\",\"judge_backend\":\"llm\",\"max_turns\":10}'\n```\n\n## Exa Blog Suite\n\nThe Exa blog suite is:\n\n```text\nsimpleqa\n2wikimultihopqa\nhotpotqa\nframes\nmusique\nbrowsecomp\n```\n\nUse `benchmark_suite=\"exa\"` to run that set:\n\n```bash\nprime eval run web-search-env \\\n  -m openai/gpt-4.1-mini \\\n  -n 30 -r 1 -t 1024 -T 0.2 \\\n  -a '{\n    \"benchmark_suite\":\"exa\",\n    \"benchmark_examples_per_dataset\":5,\n    \"search_backend\":\"exa\",\n    \"judge_backend\":\"llm\",\n    \"cache_path\":\"./search_traces/exa_suite.json\"\n  }'\n```\n\n`benchmark_examples_per_dataset=5` means 5 examples from each of the 6 datasets,\nso `-n 30` runs 30 total rollouts when `-r 1`.\n\nThis selects the same public benchmark set described in the Exa post. Exact\nreported numbers still depend on the model, sampling settings, judge, seeds, and\nlive search results.\n\nFor an n=8 run, use:\n\n```bash\nprime eval run configs/eval/web-search-exa-suite-n8.toml\n```\n\nPackaged copies of the n=8 configs are also included under\n`environments/web_search_env/configs/eval/`.\n\nThat config runs the six-benchmark suite as one eval. To print per-benchmark\nscores from the saved `results.jsonl`, run:\n\n```bash\npython environments/web_search_env/scripts/summarize_by_benchmark.py \\\n  environments/web_search_env/outputs/evals/<run-dir>/<run-id>/results.jsonl\n```\n\nUse `configs/eval/web-search-exa-suite-n8-by-benchmark.toml` instead if you want\nPrime's own final summary to have one eval section per benchmark.\n\n## Exa Training Setup\n\nUse `exa_training_setup=true` for the training and reward design described in\nthe Exa post. It selects MuSiQue and HotpotQA for training, uses the\nSimpleQA-style LLM grader as the terminal binary reward, applies a `-0.25`\ncontext-limit penalty, and sets the eval side to the six after-train benchmarks:\n\n```text\nmusique\nhotpotqa\n2wikimultihopqa\nframes\nbrowsecomp\nsimpleqa\n```\n\nHosted training config:\n\n```bash\nprime train configs/rl/web-search-exa-training.toml \\\n  --env-var EXA_API_KEY \\\n  --env-var OPENAI_API_KEY\n```\n\nThe packaged copy is also under\n`environments/web_search_env/configs/rl/web-search-exa-training.toml`.\n\n## Common Args\n\n- `eval_dataset`: dataset name or comma-separated list\n- `train_datasets`: training dataset name or comma-separated list\n- `training_suite`: use `\"exa\"` for MuSiQue + HotpotQA training data\n- `exa_training_setup`: use the Exa training/reward/eval defaults\n- `benchmark_suite`: use `\"exa\"`, `\"hle\"`, or `\"widesearch\"`\n- `benchmark_examples_per_dataset`: per-dataset cap for suite runs\n- `train_examples_per_dataset`: per-dataset cap for training suite data\n- `search_backend`: `fixture`, `exa`, `parallel`, `serpapi`, or `serper`\n- `result_count`: search results per tool call, default `5`\n- `snippet_chars`: max snippet characters per result, default `2000`\n- `search_max_qps`: optional per-process request cap for live search providers\n- `search_max_retries`: retry count for retryable search failures\n- `search_initial_jitter_seconds`: random delay before each rollout's first live search\n- `group_efficiency_bonus`: optional group-level bonus for correct rollouts that use fewer searches/turns\n- `group_efficiency_search_weight`: search-count share of the efficiency bonus\n- `group_efficiency_turn_weight`: turn-count share of the efficiency bonus\n- `group_efficiency_length_weight`: response-length share of the efficiency bonus\n- `judge_backend`: `simpleqa`, `llm`, `normalized`, `exact`, or `substring`\n- `judge_model`: judge model for `judge_backend=\"simpleqa\"` or `judge_backend=\"llm\"`\n- `max_turns`: max assistant/tool turns, default `10`\n\n\n\n\n# Results Record\n\n## Qwen3.5-35B-A3B + Exa\n\n```\nprime eval run configs/eval/web-search-exa-suite-100x3-local-vllm.toml\n```\n\n   benchmark          examples    rollouts    reward    pass@1    pass@k\n  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━  ━━━━━━━━\n   2wikimultihopqa         100         300     0.930     0.930     0.930\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   browsecomp              100         300     0.137     0.130     0.210\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   frames                  100         300     0.740     0.700     0.820\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   hotpotqa                100         300     0.747     0.750     0.800\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   musique                 100         300     0.510     0.540     0.650\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   simpleqa                100         300     0.847     0.870     0.940\n\n  Overall reward: 0.6517.\n\n\n## Qwen3-4B + Exa\n\n```\nprime eval run configs/eval/web-search-exa-suite-100x3-qwen3-4b-local-vllm.toml\n```\n\n   benchmark          examples    rollouts    reward    pass@1    pass@k\n  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━  ━━━━━━━━\n   2wikimultihopqa         100         300     0.567     0.560     0.730\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   browsecomp              100         300     0.043     0.050     0.100\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   frames                  100         300     0.377     0.390     0.550\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   hotpotqa                100         300     0.560     0.570     0.660\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   musique                 100         300     0.217     0.210     0.340\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   simpleqa                100         300     0.757     0.770     0.830\n\n  Overall reward: 0.420.\n\n\n## Qwen3.5-4B + Exa\n\n```\nprime eval run configs/eval/web-search-exa-suite-100x3-qwen35-4b-local-vllm.toml\n```\n\n   benchmark          examples    rollouts    reward    pass@1    pass@k\n  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━  ━━━━━━━━\n   2wikimultihopqa         100         300     0.917     0.890     0.960\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   browsecomp              100         300     0.093     0.090     0.150\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   frames                  100         300     0.677     0.680     0.830\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   hotpotqa                100         300     0.717     0.710     0.830\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   musique                 100         300     0.387     0.380     0.560\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   simpleqa                100         300     0.850     0.880     0.920\n\n  Overall reward: 0.6067.\n\n\n## Qwen3.5-4B + Serper\n\n```\nprime eval run configs/eval/web-search-exa-suite-100x3-qwen35-4b-serper-local-vllm.toml\n```\n\n   benchmark          examples    rollouts    reward    pass@1    pass@k\n  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━  ━━━━━━━━\n   2wikimultihopqa         100         300     0.867     0.880     0.920\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   browsecomp              100         300     0.063     0.040     0.120\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   frames                  100         300     0.623     0.630     0.790\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   hotpotqa                100         300     0.630     0.610     0.740\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   musique                 100         300     0.363     0.330     0.510\n  ─────────────────  ──────────  ──────────  ────────  ────────  ────────\n   simpleqa                100         300     0.900     0.900     0.940\n\n  Overall reward: 0.5744.\n\n\n\n# Providers\n\n## Parallel\n\nRate Limit: 600 requests per minute\nSearch Pricing: $5/1000 requests or $0.0005 per req\nExtract Pricing: $1/1000\n\n## Exa\n\nRate Limits: 10 requests per second\nSearch Pricing: $7/1000 requests or $0.007 per req\nExtract Pricing: $1/1000\n\n\n\n","encoding":"utf-8","truncated":false,"total_bytes":13924},"status":null}