{"data":{"kind":"file","path":"README.md","version_id":"jqpk9e8hehbp40vx66gs7u81","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1375,"modified_at":"2026-09-16T12:25:28.068000","content_hash":"bbed8e4005068ff4a4bbcc9ef0a56f6cc89e9652ad1face7bf6c272feec69014"},"entries":[],"content":"# frontierscience\n\nPhD-level science-olympiad QA (physics / chemistry / biology) from OpenAI's\nfrontierscience test split. Single-turn; the dataset embeds its own instruction\nand each answer (after the last `FINAL ANSWER` marker) is graded by an LLM judge\n(`FrontierScienceJudge`, default `openai/gpt-5.4-mini`) that rules it equivalent\nto the reference or not.\n\n## Taskset\n\n- **Source:** [`openai/frontierscience`](https://huggingface.co/datasets/openai/frontierscience) (`test` split)\n- **Size:** 160 tasks\n\n## Notes\n\n- The default model and the LLM judge (default `openai/gpt-5.4-mini`) are served by Prime inference, so a Prime API key is required (`PRIME_API_KEY` or `~/.prime/config.json`).\n- The `rlm` harness needs a Linux container runtime (`prime` or `docker`); it cannot run on the local `subprocess` runtime, since `flock` is unavailable on macOS.\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-08-30: Standardized binary verdict parsing and failure handling on `ReferenceJudge` while retaining final-answer extraction and the benchmark prompt.\n- 2026-07-10: Initial task-centric (`verifiers.v1`) port of the v0 `frontierscience` env. Rewards and metrics live on the `Task`; the judge model/knobs moved to `--env.taskset.task.judge.*`. Requires `verifiers>=0.2.0` and Python `>=3.11`.\n","encoding":"utf-8","truncated":false,"total_bytes":1375},"status":null}