{"data":{"kind":"file","path":"README.md","version_id":"pl32t3fhic139tpz8vzlpx7g","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1489,"modified_at":"2026-09-13T04:06:06.936000","content_hash":"d412ddb496e2f27c0395583164e524077ded44397e0d31344678733fc412cdaf"},"entries":[],"content":"# mmmu-pro\n\nMMMU-Pro multimodal multiple-choice questions — the harder, vision-centric revision of MMMU. Ports the official CoT evaluation setting with upstream prompts and answer extraction.\n\n## Taskset\n\n- **Source:** [`MMMU/MMMU_Pro`](https://huggingface.co/datasets/MMMU/MMMU_Pro) (`test` split, 1730 rows per variant)\n- **Variants (`--env.taskset.variant`):**\n  - `standard` (default): question text + lettered options + referenced images; `--env.taskset.num-options` picks the 10-option (default, MMMU-Pro's headline setting) or 4-option dataset\n  - `vision`: a single screenshot containing the question and options, with only the upstream vision instruction as text\n- **Subjects:** all by default, or one via `--env.taskset.subject` (e.g. `Math`)\n- **Prompts:** upstream CoT prompts verbatim ([`prompts.yaml`](https://github.com/MMMU-Benchmark/MMMU/blob/main/mmmu-pro/prompts.yaml)); prompt assembly mirrors the upstream inference scripts (text block first, images appended in `<image N>` marker order)\n- **Scoring:** upstream [`parse_multi_choice_response`](https://github.com/MMMU-Benchmark/MMMU/blob/main/mmmu-pro/evaluate.py) extraction, exact match on the option letter\n\n## Deviations from upstream\n\n- When no answer can be parsed from the response, upstream scores a *random* option letter; this port scores 0 instead so rewards are deterministic (upstream's fallback adds ~1/num_options expected accuracy on unparsable responses).\n\n## Changelog\n\n- 2026-07-19: Initial port.\n","encoding":"utf-8","truncated":false,"total_bytes":1489},"status":null}