{"data":{"kind":"file","path":"README.md","version_id":"xqcko15g8eypzqi0dzfhmmmc","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":17633,"modified_at":"2026-07-23T03:45:28.416000","content_hash":"1f49665d44a50a2aa17d6cf6e01c58f2cd5e5a72a24944b0a6d4836d07887600"},"entries":[],"content":"# ai-law-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated AI/export-control work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Five original, fully synthetic matters (no real client facts anywhere):\nan advanced-accelerator shipment with unresolved license responsibility and red-flag customers; frontier model weights escrowed in Singapore with a Malaysian failover; a GPU reseller with diversion indicators; a foreign-parent customer reserving a frontier-scale training cluster; and a model-export review. Each matter contains seven planted issues the memo should raise — and\ndeliberately irrelevant documents a disciplined reviewer should decline to escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. By default\nthere is no AI judging AI — the grading is deterministic, published in compiled form with\nthe package, and frozen by cryptographic hash so results are reproducible and comparable.\nAn optional judge GATE (v0.2.0, off unless you pass `judge_model`) can only *subtract*\ndeterministic credit for findings whose treatment a strict judge cannot confirm against a\nverbatim memo quote — it never adds credit, so the deterministic key remains the ceiling.\n\n**What the scores tell you.** Three things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise or say something it shouldn't have?* (`false_flag_weight`\nand `must_not_say_weight`, both scored by default as of v0.1.1). Quoted source text earns\nnothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\nAI/export-control work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine, so results are comparable across areas.\n\n## Quickstart\n\n```bash\nprime env install ai-law-legal-agent-eval\nprime eval run ai-law-legal-agent-eval -m <your-model>      # 5 tasks x 3 rollouts\nprime eval run ai-law-legal-agent-eval -m <your-model> -a '{\"task_id\":\"orion-compute-001\"}'\n```\n\nThe agent gets four tools -- `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` -- and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward; the judge, when enabled, is a remove-only GATE, not\nan additive term (an additive judge term is farmable — a model can shotgun the judge's\nlanguage to bank judge points on top of lexical points). The anchor specifications are\n**compiled from private answer keys** (never hand-written per task) and are designed to be\nfrozen by SHA-256 before any reported run. Matching is deliberately conservative --\nquoted source text is masked before scoring so pasting documents earns nothing, and a\nfinding only credits when the issue is named in its own terms *and* corroborated by\nseverity, remediation, or record-grounding evidence in the same context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nFive synthetic AI law matters, each a realistic dataroom-review delegation:\nterse partner instruction, 9-11 mixed-format documents (docx / pdf / xlsx / eml / txt),\none declared `.docx` memo deliverable, 7 planted issues, and a public rubric. All content\nis original synthetic material -- no third-party benchmark data, no real client facts.\nDocument corpora are intentionally compact in v0.1; corpus enrichment (full-length\ntechnical materials, compliance references, transaction records, multi-deliverable\ntasks) is the named path to v1.0.\n\nFour of the five tasks use the family's standard 42-criterion rubric.\n`orion-compute-001` legitimately ships 48 criteria per the factory manifest: extra\ncitation and overall-conclusion criteria reference multiple planted issues at once, and\na few pure fact-recital criteria reference no single issue. This does not change the\nscored reward shape. Every AI-law task, including Orion, still has exactly 7 planted\nfindings scored by the same `finding_recall` (0.6) and `deliverable_complete` (0.4)\nweights; the extra criteria only add resolution to the weight-0\n`criterion_pass_rate_proxy` telemetry.\n\n| scenario_id | scenario_slug | title | deliverable | instruction |\n| --- | --- | --- | --- | --- |\n| ai-cloud-training-004 | cloud-training-004 | Prepare Cloud Training Access Controls -- Risk Memo | cloud-training-risk-memo.docx | Review the attached documents against the compliance reference and prepare a cloud training access memo. Output: cloud-training-risk-memo.docx. |\n| ai-frontier-weights-002 | frontier-weights-002 | Review Frontier Model-Weight Escrow -- Risk Memo | model-weight-risk-memo.docx | Review the attached documents against the compliance reference and prepare a model-weight export risk memo. Output: model-weight-risk-memo.docx. |\n| ai-gpu-reseller-003 | gpu-reseller-003 | Assess Indirect GPU Reseller Diligence -- Risk Memo | reseller-risk-memo.docx | Review the attached documents against the compliance reference and prepare a reseller diligence risk memo. Output: reseller-risk-memo.docx. |\n| ai-model-export-005 | model-export-005 | Analyze AI Lab Acquisition Model-Export Issues -- Risk Memo | ai-acquisition-risk-memo.docx | Review the attached documents against the compliance reference and prepare an AI acquisition export memo. Output: ai-acquisition-risk-memo.docx. |\n| ai-orion-compute-001 | orion-compute-001 | Analyze Project Orion Export-Control Risk -- Risk Memo | export-control-risk-memo.docx | Review the attached documents against the compliance reference and prepare an export-control risk memo. Output: export-control-risk-memo.docx. |\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `source` | `\"factory\"` | `\"factory\"` loads the bundled anchored tasks; `\"lab\"` loads LAB tasks from an explicit `tasks_root` |\n| `task_id` | `\"all\"` | All five matters, or a single slug such as `orion-compute-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `-0.15` | Weight of the distractor-escalation penalty (`false_flag_rate`, a 0..1 rate) |\n| `must_not_say_weight` | `-0.10` | Weight of the forbidden-content penalty (`must_not_say_rate`, a 0..1 rate normalized by the task's must_not_say spec count) |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | When set (factory mode), activates the `finding_recall` judge GATE (v0.2.0); requires `JUDGE_API_KEY` (or `judge_api_key_var`). Unset = fully deterministic, byte-identical to `<=0.1.1` |\n| `judge_gate` | `True` | Whether a configured `judge_model` acts as the gate. `False` disables the gate (factory scoring stays deterministic even with a judge_model set) |\n| `judge_strict` | `True` | On judge failure (API/parse error after retries), withhold this rollout's finding credit (`finding_recall -> 0`) and flag `judge_error=1`. `False` keeps the lexical credit, still flagged |\n| `judge_votes` | `3` | Majority-vote calls per rollout; an issue is confirmed only if a majority of quote-verified votes confirm it |\n| `judge_base_url` / `judge_api_key_var` | `None` / `\"JUDGE_API_KEY\"` | Judge client endpoint and the env var holding its key (only read when `judge_model` is set) |\n| `tasks_root` | `None` | In factory mode, override the bundled task directory with the same layout; in LAB mode, required and must point at your own LAB checkout |\n\nNo environment variables are required by default.\n\n## LAB mode\n\n`source=\"lab\"` measures workflow discipline: did the agent persist every declared\ndeliverable under its exact filename with substantive content. It does not measure issue\ncoverage because LAB tasks do not ship with a compiled answer key, so there is no\n`finding_recall`, `false_flag_rate`, or similar anchored metric in this mode. Every LAB\nmetric carries a `lab_` prefix so LAB workflow telemetry is never confused with or\naveraged alongside factory metrics.\n\nLAB mode requires `tasks_root`; no default or bundled LAB tasks ship in the package. The\nloader discovers `tasks_root/<category>/<task-id>/task.json`, with source files in the\nsibling `documents/` directory. Each `task.json` must contain `instructions` as a string\nand `deliverables` as a dict whose keys are the declared filenames, for example\n`{\"memo.docx\": \"memo.docx\"}`. The internal row slug uses `category/task-id` to keep tasks\nfrom different categories distinct.\n\n## Baseline results\n\nBaseline runs are pending. No paid eval has been run for this AI-law package in this\nbuild.\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly; `finding_recall`\nis the discriminative axis and is strict by construction -- credit requires engaging with\nthe specific planted issue, not mentioning its topic. `false_flag_rate > 0` indicates the\nagent escalates noise. `transcript_recall > finding_recall` indicates findings that never\nmade it into the saved deliverable (a workflow-discipline failure, not a knowledge one).\nRollout-to-rollout variance is real at this scale; the default 3 rollouts per task are a\nminimum for stable comparisons.\n\n## Provenance\n\nTask content and answer keys are original work from a validated synthetic-task factory\n(35 tasks across 7 practice areas; this package ships the AI law pocket). Anchor\ncompilation lineage is recorded in each task's `anchors.json` provenance block and in the\nstamped module header. This build has not been frozen; freeze SHA completion is a\nseparate manual step per SPEC section 6. See `NOTICE` for attribution details.\n\nThis is one environment in a per-practice-area family; contracts, corporate governance,\nmergers & acquisitions, cybersecurity & privacy, employment law, and crypto law pockets\nfollow the same engine and scoring contract.\n\n## Known issues\n- **`finding_recall` is pure keyword/phrase-proximity matching, not semantic grading.** A\n  \"surgical shotgun\" built purely from each finding's own published vocabulary — not copied\n  from any source document, zero legal reasoning, zero citations — scores **1.0000, the maximum\n  possible reward**, matching or beating genuine analysis.\n- **`must_not_say_rate` is shallowly gameable.** The negation check only scans for a negation\n  word anywhere in the matched window, not whether it grammatically negates the claim —\n  scattering \"not\" near (not attached to) trigger vocabulary drops the penalty ~86% (rate\n  1.0 → 0.14) without any real negation.\n- **Sign-convention warning:** despite the \"same engine and scoring contract\" line above,\n  `employment-law-legal-agent-eval`'s `false_flag_weight`/`must_not_say_weight` are positive\n  magnitudes subtracted internally — the opposite convention from this env's weights (negative\n  numbers passed directly to the rubric). Do not reuse weight kwargs across siblings.\n- v0.1.1 fixes the wiring (must_not_say now applies a real penalty by default) but does not\n  close the shotgun or evasion gaps above. Do not train unsupervised against this reward as\n  evidence of genuine legal-analysis quality.\n- **v0.2.0 partially addresses the shotgun hole, but only when you opt in.** Passing a\n  `judge_model` activates a remove-only GATE over `finding_recall`: a strict, injection-fenced\n  judge must confirm each lexically-credited finding is a genuine, matter-specific treatment\n  and return a verbatim memo quote (checked mechanically) or that finding loses credit. The\n  gate is validated at the *pattern* level (ported from `agent-liability-gym`, which measured\n  0 false-positives on 19 exploit findings / 0 false-negatives on 30 honest memos);\n  **Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini policy, n=10 rollouts):\n  judge_gated 10/10, judge_error 0/10, gate strictly remove-only, reward algebra exact;\n  removals spot-verified as genuine lexical false-positives. See outputs/evals/ for the\n  run.** The default (no `judge_model`) path is unchanged and still shotgun-gameable. The gate cannot rescue `value_match`-style holes it\n  has no signal for, and it does nothing in LAB mode (no answer key to gate against).\n\n## Changelog\n\n- **v0.3.0** — **Negative-control passthrough fix (family stop-ship 2026-07-22).**\n  Negative-control findings (`kind=\"negative_control\"`, e.g. `ISS-007`) were lexically\n  earnable in `finding_recall`: a memo that merely uttered the planted \"negative control\"\n  jargon plus a couple of distractor evidence tokens was CREDITED, banking +1/n_findings of\n  `finding_recall` (measured **≈+0.086** net default reward at the 0.6 `finding_recall`\n  weight — `false_flag`/`must_not_say` do not offset it). In gated mode that NC credit sat\n  in the gate's `passthrough` set (the judge is never asked about NCs), so it was\n  un-removable. `finding_recall_details` now NEVER credits an NC: the NC stays in the\n  denominator (max recall = n_substantive / n_findings) and its lexical match is recorded as\n  telemetry only (`nc_matched_no_credit`, `nc_lexical_match_count`, `substantive_max_recall`).\n  `all_pass` now fires at `substantive_max_recall` rather than a literal `1.0` (every matter\n  ships exactly one NC, so the old `==1.0` gate was otherwise unreachable and, pre-fix, could\n  only be satisfied by tripping the NC trap). **This changes r-mode scoring: pre-v0.3.0\n  scores are comparable only for rollouts where no NC was lexically credited. Honest, NC-free\n  memos are byte-identical; any rollout that had banked NC credit now scores 1/n_findings\n  lower on `finding_recall`.** Mirrors the validated `agent-liability-gym` NC treatment.\n  Regression tests: `tests/test_nc_passthrough.py`.\n- **v0.2.1** — Doc-only. Replaced the pattern-ported placeholder validation status on the\n  `finding_recall` judge gate with the 2026-07-22 live judge-gate validation result. No\n  code changes.\n- **v0.2.0** — Judge GATE for `finding_recall` (opt-in). When `judge_model` is set\n  in factory mode, the judge is wired in as an AND-gate that can only REMOVE\n  lexically-credited findings, never add them; each confirmation is verified\n  deterministically against a verbatim memo quote (a hallucinated quote loses the\n  slot), runs as a temperature-0 majority vote (`judge_votes`, default 3) with an\n  in-process cache, and fails loud (`judge_error=1`, credit withheld under the\n  default `judge_strict`) rather than silently scoring ungated. The legacy weight-0\n  factory JudgeRubric (\"telemetry\") was REMOVED and replaced by the gate; the LAB\n  weight-0 monitor is retained (LAB has no answer key to gate). New params\n  (`judge_gate`, `judge_strict`, `judge_max_retries`, `judge_votes`) are threaded\n  through `load_environment`/`build_environment`. New 0-weight telemetry metrics:\n  `finding_recall_lexical` (pre-gate recall), `judge_gated`, `judge_error`.\n  **Default (no `judge_model`) scoring is byte-identical to v0.1.1** — reward is\n  still `0.4*deliverable_complete + 0.6*finding_recall - 0.15*false_flag_rate -\n  0.10*must_not_say_rate`, and the existing test suite passes unchanged.\n  Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini policy, n=10 rollouts):\n  judge_gated 10/10, judge_error 0/10, gate strictly remove-only, reward algebra\n  exact; removals spot-verified as genuine lexical false-positives. See\n  outputs/evals/ for the run.\n- **v0.1.1** — Reward-scoring fix. `must_not_say_hit_count` returned a raw,\n  unnormalized integer instead of a 0..1 rate, and no `must_not_say_weight`\n  kwarg existed to score it at all (hardcoded weight 0.0), so forbidden-content\n  language never affected reward regardless of how it was configured. Fixed:\n  the metric is now `must_not_say_rate_details`, normalized by each task's\n  own must_not_say spec count (raw hits are still reported separately as\n  `must_not_say_hits` for diagnostics), a `must_not_say_weight` kwarg was\n  added and threaded through `load_environment`/`build_environment`\n  (default `-0.10`), and `false_flag_weight`'s default changed from `0.0`\n  to `-0.15` so distractor escalation is scored out of the box. Net effect:\n  `reward` is no longer exactly `0.4*deliverable_complete +\n  0.6*finding_recall` under default arguments — it now also subtracts\n  `0.15*false_flag_rate + 0.10*must_not_say_rate`. Scores from `<=0.1.0` runs\n  that had any nonzero `false_flag_rate` or `must_not_say_hits` are not\n  directly comparable to `>=0.1.1` runs under default arguments; re-run for\n  an apples-to-apples comparison if either metric fired.\n- **v0.1.0** — Initial release.\n","encoding":"utf-8","truncated":false,"total_bytes":17633},"status":null}