{"data":{"kind":"file","path":"README.md","version_id":"yhic13021zqfitfv74c07cbk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1948,"modified_at":"2026-09-13T04:06:06.929000","content_hash":"29c51e53dae4d8affef75b6146015d0eb939611c5b56142f888715f9b4895042"},"entries":[],"content":"# longpdfs\n\nLong-document Q&A tasks where an RLM agent is given several long PDF-extracted documents on disk (too large to all fit in context) and must answer questions about each. Answers are scored by exact match (case-insensitive) short-circuiting to an LLM judge, with reward from the per-document pass rate plus a perfect-rollout bonus.\n\n## Taskset\n\n- **Source:** [`PrimeIntellect/longpdfs`](https://huggingface.co/datasets/PrimeIntellect/longpdfs) (private HF dataset)\n- **Size:** Procedurally generated — 700 tasks by default, each bundling 10 documents drawn with replacement from the source dataset (train split).\n\n## Notes\n\n- The Q&A pairs are LLM-generated and then quality-filtered (questions answerable without the document are dropped, plus a consistency check).\n- Documents default to English-only (`eng_Latn`); pass `--env.taskset.languages all` for every language.\n- The judge is strict: non-committal answers count as wrong.\n\n## Changelog\n\n- 2026-09-03: Restore default solver network access by reverting the `network_allow=[]` default-deny policy introduced in #780; training rollouts need outbound network.\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-08-30: Standardized per-question semantic equivalence checks on `ReferenceJudge` while retaining answer-file fan-out, exact-match short-circuiting, and document-level rewards.\n- 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the `Task` (a `TaskData` row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from `--env.taskset.*` to `--env.taskset.task.*`. Requires `verifiers>=0.2.0` and Python `>=3.11`.\n- 0.0.1\n  - Create environment\n  - Add LLM-judge scoring (`document_pass_rate` + `full_pass` rewards, `question_accuracy`/`answer_rate` metrics)\n  - Default to English-only documents; add `train`/`test` split\n","encoding":"utf-8","truncated":false,"total_bytes":1948},"status":null}