{"data":{"kind":"file","path":"README.md","version_id":"vqo1ym2tcvjeodgl9pgkb09j","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9894,"modified_at":"2026-08-06T21:23:05.977000","content_hash":"2cfd3ddbda6cb29b8759b7ca604e919938d8c50de3b5712fa856720dc3276865"},"entries":[],"content":"# legal-qa\n\nQuestions that were put to American government bodies, graded against the\nanswers those bodies published.\n\n- **State attorney general opinions.** A legislator or an agency asks whether\n  something is lawful; the attorney general answers.\n- **State tax rulings.** A taxpayer describes a transaction and asks whether it\n  is taxable; the revenue department answers.\n- **Bar ethics opinions.** A lawyer asks whether conduct is permitted; the\n  bar's ethics committee answers.\n\nNobody involved in the grading wrote the answer key. A state attorney general\ndid, sometimes decades before anyone thought of scoring a model against it.\n\n**1,500 tasks, 500 per corpus, 75 issuing bodies** covering 48 states, the\nDistrict of Columbia and the American Bar Association, from 1952 to 2026. This\nis the open slice of a\nlarger set, published so the grader can be checked and the numbers reproduced\nwithout asking anyone for anything.\n\n## Why law\n\nLaw is the standard example of a domain that resists verification. Code is\nverifiable because tests run, maths because answers check; the flagship legal\nbenchmarks fall back on rubrics scored by an LLM judge, and the published\nagreement rate for three judges on the same rubric is 81%.\n\nHere the bottom line, the controlling provisions and the figures are all\ncheckable by string comparison against what the issuing body wrote. No judge\nmodel is involved at any point.\n\n## Signals\n\n    verdict            did the answer reach the same bottom line   (reward, weight 1.0)\n    authority          did it name a provision the body relied on  (reward, weight 0.5)\n    authority_recall   share of the listed provisions named        (metric)\n    figures            share of the deadlines and amounts stated   (metric)\n    committed          did it state a bottom line of any kind      (metric)\n    hedged             said \"it depends\" where the body did not    (metric)\n\nRecall is reported and not rewarded on purpose. These sources list everything\nan opinion touched, a median of four provisions and six for an attorney general\nopinion, while a correct answer usually turns on one or two. Rewarding recall\nagainst that denominator pays for citation padding.\n\n## Usage\n\n```bash\nprime env install ezel/legal-qa\n```\n\n```bash\n# everything\nvf-eval legal_qa\n\n# one corpus, first 100 tasks\nvf-eval legal_qa --taskset.corpus tax --taskset.limit 100\n```\n\n`corpus` takes `ag`, `tax`, `ethics` or `all`. `limit` caps the task count.\n\n## A first result\n\n120 tasks, `claude-opus-5`, no retrieval, no examples, answered in an empty\ndirectory with every tool disabled:\n\n| corpus | agrees with the issuing body | best constant-answer baseline | names a provision the body relied on |\n|--------|------------------------------|-------------------------------|--------------------------------------|\n| ethics | 62.5% | 46.2% | 80.0% |\n| tax    | 55.0% | 51.8% | 92.5% |\n| AG     | 55.0% | 49.4% | 80.0% |\n| **all** | **57.5%** | **54.2%** | 84.2% |\n\n**Read the interval before the score.** 57.5% on 120 tasks is 48.6 to 66.0 at\n95%, which does not separate from the 54.2% baseline. This sample says the\nmodel is not clearly better than answering \"no\" to everything; it does not yet\nsay by how much. Run more tasks if you want a number to quote.\n\n**The finding that does survive the sample size is the hedging.** On 52 of the\n120 the model answered \"It depends\", and in most of those the issuing body gave\na flat yes or no. That is 43.3%, an interval of 34.8 to 52.3, and it is not\nnoise. Where the body itself hedged the key already accepts \"it depends\", so\nthese are questions that had an answer and did not get one:\n\n- A West Virginia county commission asked whether it could use eminent domain\n  for a particular taking. The attorney general said yes. The model said it\n  depends.\n- A South Carolina taxpayer asked whether a two-building expansion qualified\n  for a five-year exemption. The department said yes. The model said it\n  depends.\n- A Montana lawyer asked whether one lawyer could represent both sides. The\n  committee said no. The model said it depends.\n\nIt is not that the model does not know the law. It names a controlling\nprovision on 84.2% of answers, and 92.5% on tax. It declines to say what the\nprovision requires.\n\nThe baselines are printed for a reason. Tax rulings say no more often than yes,\nso a model that learned nothing but that habit already clears half the set, and\na verdict number that fails to beat the baseline is measuring the base rate.\n\n## Things worth knowing before you trust the number\n\n**The as-of date is load-bearing.** These bodies answered under the law as it\nstood when they wrote, so a 1993 tax ruling is graded against the 1993 code and\nevery prompt says so. Leaving that out turns every superseded rule into a\nscored error.\n\n**Where the issuing body hedged, the key accepts either answer.** \"Yes, if the\nagreement is structured under § 31-7-14\" and \"Probably not\" both accept the\nbody's own answer or \"it depends\", because marking a model wrong there measures\nformatting rather than law. 259 of the 1,500 keys, about one in six.\n\n**The verdict is read off the first word and nowhere else.** The prompt asks for\nit there. Digging through the body for a verdict the model declined to state is\nhow a grader starts inventing agreement.\n\n**Constitutional provisions are dropped from the key.** \"Tex. Const. art. XVI,\n§ 61\" reduces to \"61\", which any answer mentioning any section 61 would be\ncredited for. Dropping these loses real authority; keeping them handed out\ncredit nobody earned.\n\n**These documents are published.** A model may have seen them. That matters for\nreading a benchmark score and does not matter for training data. It has not\nsaturated: a frontier model disagrees with the issuing body on four answers\nin ten.\n\n## Checking it yourself\n\nEvery task carries `source_url`, the issuing body's own page for the document\nit came from. So you do not have to take anybody's word about who reviewed\nthis, including ours. Pull any item, open its link, and read the opinion:\n\n```python\nfrom legal_qa import LegalQATaskset, LegalQATasksetConfig\ntask = next(iter(LegalQATaskset(LegalQATasksetConfig(id=\"legal_qa\", limit=1)).load()))\nprint(task.data.question, task.data.answer_key[\"verdict\"], task.data.source_url)\n```\n\n75% of the links go straight to the document; the rest go to the index page it\nis listed on. The link is deliberately not in the prompt: the model gets the\nquestion and nothing else.\n\nThat is the intended way to audit this. A signed statement that some attorney\nchecked it is worth less than being able to check twenty items yourself in an\nafternoon.\n\n## How accurate the keys are: 99 of 100\n\nThe key is only worth what its accuracy is, so that was measured rather than\nasserted. A random sample of 100 tasks, stratified across the three corpora,\nwas read by hand against the original documents. Not spot-checked: the issuing\nbody's own text next to the question and the key, one at a time.\n\n**99 of 100**, a 95% confidence interval of 94.6 to 99.8.\n\nThe single miss shows the shape of what remains. A Colorado TABOR question asks\nwhether the book value of specific rebuilt highway lanes counts toward a\nspending limit. The key says \"it depends\", because the summary states the\ngeneral rule that way, where the opinion's answer for those particular lanes is\na flat no. Nothing mechanical catches that; the summary is accurate and the\nquestion is narrower than the summary. Expect a residue of that kind at roughly\none percent.\n\nThe sample that produced this number was drawn separately from the one used to\ndevelop the reader, so it measures the keys rather than the work that shaped\nthem.\n\n## How the keys are read\n\nNot by counting. A misread verdict does not move a total, so totals cannot find\none. Every rule in the reader was checked by printing its decisions next to the\nquestions they came from and reading them, and every rule that exists is\nanswering a real document that would otherwise be read backwards. A few of the\ndistinctions that turn out to matter:\n\n- **\"Generally no.\" is a no.** A hedge takes either polarity, and reading the\n  hedge before the polarity inverts the answer.\n- **A negated prohibition is a permission.** \"Rule 1.5(d)(1) does not bar a\n  contingent fee\" means the lawyer may, unless the question was itself asking\n  whether a rule bars it, in which case it means exactly what it says.\n- **A denied bar is not a refusal.** \"No conflict\", \"No violation\", \"no per se\n  conflict\" all mean the thing asked about is permitted. Same four words,\n  opposite verdicts, and only the shape of the question separates them.\n- **A compilation year is not a section.** \"NMSA 1978\" and \"R.R.S. 1943\" name\n  the code, not a provision, so a four-digit token counts only where a section\n  symbol precedes it. That keeps 42 U.S.C. § 1983 and 28 U.S.C. § 1875.\n- **A positive copula is a premise.** These summaries open on their reasoning,\n  and \"transferring a lien is a ministerial act\" does not answer the question\n  that was put. Only words granting or withholding permission are read.\n\nOne failure mode is excluded rather than papered over. Where the verdict has to\nbe inferred from the holding rather than read off the first word, the summary's\nsubject is often not the question's subject: \"Can a lawyer mark up a contract\nattorney's rate without telling the client?\" answered \"the engaging lawyer must\ndisclose\" reads as a yes and means no. Those keys agree with a model 18% of the\ntime, below chance. Only first-word keys ship here.\n\nThe builder, the audit tool and 67 regression cases are in the source\nrepository: <https://github.com/ezel-ai/legal-qa-bench>\n\n## Licence\n\nMIT for the code. The underlying opinions, rulings and ethics opinions are\npublic documents of US state and federal bodies. The plain-English questions\nand answers were written by Ezel from those documents.\n","encoding":"utf-8","truncated":false,"total_bytes":9894},"status":null}