{"data":{"kind":"file","path":"README.md","version_id":"tzh4zofc2vezwgkusxu544js","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3368,"modified_at":"2026-08-22T02:34:35.850000","content_hash":"2a76f9c1f6c865351eedd3ef0afd45bb55771508fc38408278e271fa8e9ee31e"},"entries":[],"content":"# grc-mapping\n\nCompliance control mapping across **686 frameworks**, graded against a live\nknowledge graph with **38,707 published refutations**.\n\n## What the model is asked to do\n\nFour families, drawn from released framework pairs that a person reviewed and\nsigned off:\n\n1. **Single-pair mapping.** Given a control in framework A, name its counterpart\n   in framework B. Sometimes there is exactly one; sometimes several, and the\n   question says which is being asked.\n2. **Multi-framework crosswalk.** The same control against several frameworks at\n   once, where coverage is genuinely asymmetric.\n3. **Provenance citation.** The verdict plus the source document that grounds it.\n   A citation that does not ground the claim scores zero even when the verdict is\n   right.\n4. **Distractor rejection.** Accept or reject a candidate mapping. **The\n   distractors are real refuted mappings**, each one plausible enough that a\n   mapping process proposed it before it failed review.\n\n## Why the hard negatives are the point\n\nMost benchmarks build distractors by sampling something topically similar, which\nteaches a model to spot topic drift. These were proposed by a real mapping\nprocess and then rejected, so they are wrong in the ways compliance mappings are\nactually wrong: a policy control credited against a substantive implementation\nrequirement, a control whose title reads familiar while its body requires\nsomething else, the same coverage counted twice through a parent and its\nsubclause, a data-type mismatch.\n\nWe know because we made them. An automated mapping path of ours was re-judged by\nhand and 56% of its output was refuted.\n\n## Scoring\n\nDeterministic. No model judges anything.\n\nReward is one authenticated call to the compliance graph, which derives the\ncorrect answer at scoring time. F1 over reference sets, so partial answers score\npartially and padding is penalised through precision. Reference matching\ntolerates a framework prefix at a token boundary, so `AC-11` matches\n`NIST800-AC-11` while `AC-1` does not.\n\n**There is no answer key in this repository.** `TaskData` carries the question and\nnothing else. Answers are never stored, including by us, so this package can be\npublished openly. It also means a taskset stays correct as the graph is\ncorrected rather than grading against a stale key.\n\n## Running it\n\n```bash\nexport TAS_SCORING_KEY=...   # scoped to this taskset, from The Art of Service\nuv run eval grc-mapping\n```\n\nThe key is scoped: it can score this taskset and nothing else. It cannot page the\nlicensed endpoints and cannot score another taskset, so a leaked training key is\nworth exactly one taskset.\n\nThe questions are public and need no key:\n`https://api.theartofservice.com/api/score/tasksets/grc-mapping-v1`\n\n## What a reviewer can check without asking us\n\n**Released pairs**, with coverage and rejection counts:\n`api.theartofservice.com/api/agent/crosswalk/pairs`\n\n**The refutations themselves**:\n`api.theartofservice.com/api/agent/crosswalk/refuted`\n\n**The benchmark**: `compliance.theartofservice.com/benchmark`\nSame model, same 144 questions, with and without the graph. 63.2% to 98.9%\noutside the famous few, and the case where the graph barely helps is on the page.\n\n## Licence\n\nRetrieval and verification are licensed. No bulk export of the underlying graph\nand no training rights over it. Non-exclusive by default.\n","encoding":"utf-8","truncated":false,"total_bytes":3368},"status":null}