{"data":{"kind":"file","path":"README.md","version_id":"pf16z92zwu8xlvyqpblypshv","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":615,"modified_at":"2026-09-16T12:25:28.069000","content_hash":"8923658a649119678813f490314c848a5049acd47cec9647eaae8a0d0c7bd7ca"},"entries":[],"content":"# gpqa\n\nGPQA graduate-level multiple-choice science QA. Each task is a 4-option (A/B/C/D) science question; the model reasons and gives the answer letter, scored deterministically by extracting the letter and comparing it to the gold letter, with an LLM-judge fallback when the letter check fails.\n\n## Taskset\n\n- **Source:** [`Idavidrein/gpqa`](https://huggingface.co/datasets/Idavidrein/gpqa)\n- **Size:** 198 tasks (`gpqa_diamond`, default; `gpqa_main` has 448)\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-06-24: Initial v1 taskset.\n","encoding":"utf-8","truncated":false,"total_bytes":615},"status":null}