{"data":{"kind":"file","path":"README.md","version_id":"o7ibt9674dow751m5w62jaxp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2811,"modified_at":"2026-08-13T20:09:23.012000","content_hash":"e9d57b9def251a208a95cea86bdab1dc551e42d7750b99029da1954fd3440ead"},"entries":[],"content":"# cad-spec\n\nAn RL environment where a model reads a dimensioned engineering spec and writes\nCadQuery to satisfy it. Scoring is not text similarity: the code is executed,\nthe resulting solid is measured, and each requirement is checked against the\nmeasurement the way an inspection report would check a machined part.\n\nPart family: rectangular mounting plate, four-hole bolt pattern, parameterised\nover length, width, thickness, hole diameter and edge margin. 7 training specs,\n3 held out for eval (including the size extremes), so eval scores measure\ngeneralization, not memorization.\n\n## Reward\n\nSingle function, scale exactly [0, 1]:\n\n```\n1.0    every requirement met\nk/6    partial compliance (per-requirement PASS/FAIL)\n0.05   runnable CadQuery that satisfies nothing or fails a gate\n0.0    code that does not execute, or no code at all\n```\n\n## Why the scoring is built this way\n\nDimensional checks alone are trivially gameable — a solid block with no holes\nsatisfies every overall-dimension requirement. So the rubric has two layers.\n\n**Gates** zero the reward. Each exists because of a specific cheat:\n\n| Gate | Cheat it kills |\n|---|---|\n| `single_solid` | four loose corner tabs that share the bounding box |\n| `through_holes` | blind dimples that measure like fastener holes from above |\n| `hole_count_sane` | swiss-cheesing the plate to hit a volume target |\n| `is_plate` | a shell, hollow box, or ellipse extrusion with the right bbox |\n\n`is_plate` compares measured volume against volume predicted from the *measured*\ngeometry, not from the spec. Gating on the spec would zero any dimensional\nerror and destroy the partial credit RL needs to climb.\n\n**Requirements** give partial credit: reward is the fraction of R1–R5 met.\n\nAll of this is enforced by a 16-case adversarial harness\n(`scripts/test_rubric.py` in the source repo) covering hand-written correct,\npartially-wrong, and deliberately cheating answers — including domain cheats\nlike counterbored holes where plain through holes were specified, offset\npatterns with correct pitch, and parts built in inches.\n\n## Baseline\n\nqwen2.5-coder:1.5b (local, CPU): mean 0.24–0.42 across runs on the held-out\nspecs, with individual rollouts spanning the full range 0.0 to 1.0 and strong\nwithin-group variance — well inside the trainable band. Dominant failure mode:\nthe model completes the template correctly, then keeps writing and destroys the\npart with invented API calls. Exactly the behavior RL should remove.\n\n## Layout\n\n```\ncad_spec/measure.py      build model code, extract geometry. knows nothing about specs\ncad_spec/tasks.py        the part family, prompt template, reference solutions\ncad_spec/rubric.py       gates and requirements. the file that matters\ncad_spec/environment.py  verifiers wrapper. the only API-coupled file\n```\n","encoding":"utf-8","truncated":false,"total_bytes":2811},"status":null}