{"data":{"kind":"file","path":"README.md","version_id":"cj1in7bl44smjzh8n8h6d9f2","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1431,"modified_at":"2026-09-13T04:06:06.928000","content_hash":"728d774153b6886878af581c080b6d618ab7cffd09121ae6687c8eec6a3c26b4"},"entries":[],"content":"# longbenchpro\n\nLongBench-Pro long-context understanding tasks (8k–256k tokens): each task pairs a question with a long document uploaded to the sandbox, and an agent scans it from a persistent IPython REPL and writes its answer to a file. Scoring dispatches on the task's sub-task type to the official LongBench-Pro per-task metric (accuracy / F1 / SubEM / NDCG / pairwise accuracy), or optionally a binary yes/no LLM judge.\n\n## Taskset\n\n- **Source:** [caskcsg/LongBench-Pro](https://huggingface.co/datasets/caskcsg/LongBench-Pro) (`test` split)\n- **Size:** 1,500 tasks\n\n## Notes\n\n- By default only English examples are loaded and the `T4.*` (Summarization) tasks are excluded — their official metrics rely on model-based embeddings that are impractical here. Set `--env.taskset.language Chinese`/`all` for other languages.\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-08-30: Standardized binary verdict parsing and failure handling on `ReferenceJudge` while retaining answer-file selection and deterministic metric mode.\n- 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the `Task` (a `TaskData` row + behavior split), and task-facing config knobs (judges, tool/user placement, scoring parameters) moved from `--env.taskset.*` to `--env.taskset.task.*`. Requires `verifiers>=0.2.0` and Python `>=3.11`.\n","encoding":"utf-8","truncated":false,"total_bytes":1431},"status":null}