{"data":{"kind":"file","path":"README.md","version_id":"jpp27dqbuzzjr9mrtchjri7r","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1242,"modified_at":"2026-06-27T08:38:53.822000","content_hash":"0a8f1faa71079df7fc26386c863716f9e52cfba078a91635048c4cdf38517926"},"entries":[],"content":"# benchcad-qa\n\nRendered views of a mechanical part + one numeric question → a single number. The\nmodel is shown a 2×2 composite of four diagonal views of an industry-standard\npart and answers a question about a dimension, count, or ratio.\n\n**Reward = symmetric ratio accuracy** `min(pred, gt) / max(pred, gt)` for\ndimensions/ratios, exact match for counts/integers/yes-no. Deterministic — **no\njudge model and no CAD execution** (so this environment is lightweight: just\n`verifiers` + `datasets` + `pillow`).\n\n- Benchmark: [BenchCAD](https://github.com/BenchCAD/BenchCAD-main) · paper [arXiv:2605.10865](https://arxiv.org/abs/2605.10865)\n- Data: [`BenchCAD/BenchCAD`](https://huggingface.co/datasets/BenchCAD/BenchCAD), config `QA` (2,400 questions over 200 parts)\n\n## Usage\n\n```bash\nuv run vf-eval benchcad-qa -n 20\n```\n\n`load_environment(num_examples=None)` — set `num_examples` to evaluate the first\nN questions; omit for the full set.\n\n## Notes\n\nThe numeric scoring is vendored from the canonical scorer\n(`BenchCAD/BenchCAD-main`, `QA/scoring/qa_score.py`); that repo is the source\nof truth. See also the heavier execution-based [`benchcad-vision2code`](../benchcad-vision2code)\n(image → CadQuery program, scored by voxel IoU).\n","encoding":"utf-8","truncated":false,"total_bytes":1242},"status":null}