{"data":{"kind":"file","path":"README.md","version_id":"dpvj1vizvm2bning8v0upo3y","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":19886,"modified_at":"2026-07-23T03:45:21.912000","content_hash":"2df4fb7baa11e4064ce67619040d72a94fb4f6a3a581064700ef3bb6f453ec78"},"entries":[],"content":"# corporate-governance-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated corporate governance work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Five original, fully synthetic matters (no real client facts anywhere):\na stockholder agreement that may unlawfully constrain board authority; a controller transaction needing a properly empowered independent committee; a contested director resignation; a books-and-records demand; and a committee-process review. Each matter contains seven planted issues the memo should raise — and\ndeliberately irrelevant documents a disciplined reviewer should decline to escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. By default\nthere is no AI judging AI — the grading is deterministic, published in compiled form with\nthe package, and frozen by cryptographic hash so results are reproducible and comparable.\nAn optional judge (enabled only when you pass `judge_model`) acts strictly as a\n**remove-only GATE** on `finding_recall`: it can drop a lexically-credited finding it\ncannot confirm is a genuine, matter-specific treatment, but it can never add credit or\nraise any score. Off by default; see Environment arguments and Changelog.\nReward is a weighted sum of `deliverable_complete` (0.4), `finding_recall` (0.6),\n`false_flag_rate` (`false_flag_weight`, default **-0.15**), and `must_not_say_hits`\n(`must_not_say_weight`, default **-0.10**) — the last two are real penalties by default,\nnot telemetry-only. `must_not_say_hits` is a 0..1 rate (hits / applicable must-not-say\nspecs for the task), not a raw count, so tasks with more traps don't get disproportionate\npenalty. Because the penalty terms are negative and uncapped relative to the positive\nterms, **the composite reward can go below 0.0** for a memo that is both incomplete/\nwrong and trips a false-flag or must-not-say trap — this is intended, not a bug (see\nChangelog).\n\n**What the scores tell you.** Four things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise?* (`false_flag_rate`, penalized) — *Did it say something it\nwas explicitly told not to say — a legal-safety trap, e.g. characterizing a protected\ndisclosure as something to \"sanitize\"?* (`must_not_say_hits`, penalized). Quoted source\ntext earns nothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\ncorporate governance work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine, so results are comparable across areas.\n\n## Quickstart\n\n```bash\nprime env install corporate-governance-legal-agent-eval\nprime eval run corporate-governance-legal-agent-eval -m <your-model>      # 5 tasks x 3 rollouts\nprime eval run corporate-governance-legal-agent-eval -m <your-model> -a '{\"task_id\":\"northstar-stockholder-001\"}'\n```\n\nThe agent gets four tools -- `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` -- and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward. When a `judge_model` is supplied the judge is a\nstrict remove-only **gate** — never an additive term (an additive judge term is farmable:\nshotgun the rubric's language to bank judge points on top of lexical points). Each judge\nconfirmation is itself verified deterministically: the judge must return a verbatim memo\nquote for every finding it confirms, and the quote is checked mechanically against the\nmemo, so a judge that hallucinates evidence loses that slot. Confirmations use a majority\nvote (`judge_votes`, default 3, temperature 0) and are cached in-process so gated rewards\nare repeatable within a run. The anchor specifications are\n**compiled from private answer keys** (never hand-written per task) and are designed to be\nfrozen by SHA-256 before any reported run. Matching is deliberately conservative --\nquoted source text is masked before scoring so pasting documents earns nothing, and a\nfinding only credits when the issue is named in its own terms *and* corroborated by\nseverity, remediation, or record-grounding evidence in the same context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nFive synthetic corporate governance matters, each a realistic dataroom-review delegation:\nterse partner instruction, 9 mixed-format documents (docx / pdf / xlsx / eml / txt), one\ndeclared `.docx` memo deliverable, 7 planted issues, and a 42-criterion rubric. All\ncontent is original synthetic material -- no third-party benchmark data, no real client\nfacts. Document corpora are intentionally compact in v0.1; corpus enrichment (full-length\nboard materials, governance instruments, multi-deliverable tasks) is the named path to\nv1.0.\n\n| Task | Title | Deliverable | Instruction |\n| --- | --- | --- | --- |\n| `northstar-stockholder-001` | Review NorthStar Stockholder Agreement — Governance Memo | `stockholder-agreement-memo.docx` | Review the attached documents against the compliance reference and prepare a governance memo. Output: stockholder-agreement-memo.docx. |\n| `evergreen-controller-002` | Assess Evergreen Controller Transaction — Board Memo | `controller-process-memo.docx` | Review the attached documents against the compliance reference and prepare a board process memo. Output: controller-process-memo.docx. |\n| `pioneer-resignation-003` | Analyze Pioneer Director Resignation — Disclosure Memo | `director-resignation-memo.docx` | Review the attached documents against the compliance reference and prepare a director resignation memo. Output: director-resignation-memo.docx. |\n| `ridgeway-books-004` | Review Ridgeway Books-and-Records Demand — Response Memo | `books-records-memo.docx` | Review the attached documents against the compliance reference and prepare a books-and-records memo. Output: books-records-memo.docx. |\n| `solstice-committee-005` | Assess Solstice Special Committee Process — Process Memo | `special-committee-memo.docx` | Review the attached documents against the compliance reference and prepare a special committee memo. Output: special-committee-memo.docx. |\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `source` | `\"factory\"` | `\"factory\"` loads the bundled anchored tasks; `\"lab\"` loads LAB tasks from an explicit `tasks_root` |\n| `task_id` | `\"all\"` | All five matters, or a single slug such as `northstar-stockholder-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `-0.15` | Weight of the distractor-escalation penalty (`false_flag_rate`, 0..1) applied to reward. Negative by default since v0.1.2 |\n| `must_not_say_weight` | `-0.10` | Weight of the must-not-say legal-safety-trap penalty (`must_not_say_hits`, a 0..1 rate) applied to reward. New in v0.1.2 — previously hardcoded to 0.0 with no way to configure it |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | Optional judge. When set, enables the remove-only `finding_recall` gate (below). Requires `JUDGE_API_KEY` (or `judge_api_key_var`) only when set |\n| `judge_gate` | `True` | When a `judge_model` is set, the judge acts as a strict remove-only gate on `finding_recall` (can only drop unconfirmed lexical credit). Set `False` to restore the legacy weight-0 judge *telemetry* rubric instead |\n| `judge_strict` | `True` | On judge failure (API/parse error after retries), withhold ALL finding credit for that rollout (`judge_error=1.0`, `finding_recall→0`). `False` keeps the lexical credits, still flagged via `judge_error` |\n| `judge_votes` | `3` | Majority-vote calls per gate decision (temperature 0 first, bare fallback); an issue is confirmed only if a majority of successful votes confirm it |\n| `judge_max_retries` | `2` | Per-vote retries on transient judge errors before that vote is counted as failed |\n| `judge_base_url` | `None` | Optional OpenAI-compatible base URL for the judge client |\n| `judge_api_key_var` | `\"JUDGE_API_KEY\"` | Env var name holding the judge API key; read only when `judge_model` is set |\n| `tasks_root` | `None` | In factory mode, override the bundled task directory with the same layout; in LAB mode, required and must point at your own LAB checkout |\n\nNo environment variables are required by default. `JUDGE_API_KEY` (or the var named by\n`judge_api_key_var`) is required only when `judge_model` is set.\n\nWhen the gate is active it exposes three 0-weight telemetry metrics alongside the graded\nones: `finding_recall_lexical` (pre-gate recall), `judge_gated` (1.0 when the gate ran),\nand `judge_error` (1.0 when the judge was unavailable — a spike here across an eval means\ninfra failure, not model failure; do not read the accompanying rewards as capability\nsignal). Negative-control lexical credits are not re-adjudicated by the judge; they pass\nthrough the gate exactly as the deterministic scorer set them.\n\n## LAB mode\n\n`source=\"lab\"` measures workflow discipline: did the agent persist every declared\ndeliverable under its exact filename with substantive content. It does not measure issue\ncoverage because LAB tasks do not ship with a compiled answer key, so there is no\n`finding_recall`, `false_flag_rate`, or similar anchored metric in this mode. Every LAB\nmetric carries a `lab_` prefix so LAB workflow telemetry is never confused with or\naveraged alongside factory metrics.\n\nLAB mode requires `tasks_root`; no default or bundled LAB tasks ship in the package. The\nloader discovers `tasks_root/<category>/<task-id>/task.json`, with source files in the\nsibling `documents/` directory. Each `task.json` must contain `instructions` as a string\nand `deliverables` as a dict whose keys are the declared filenames, for example\n`{\"memo.docx\": \"memo.docx\"}`. The internal row slug uses `category/task-id` to keep tasks\nfrom different categories distinct.\n\n## Baseline results\n\n| Model | Setup | Reward | Finding recall | Deliverables saved | False flags |\n| --- | --- | ---: | ---: | ---: | ---: |\n| | gpt-5.5 | 5 tasks × 3 rollouts | 0.697 | 0.495 | 15/15 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.457 | 0.095 | 15/15 |\n\nThe strongest frontier separation in the family (5x the 30B's recall). Credited findings\nwere hand-audited against the answer keys — zero over-credit. | - | - | - | - | - |\n\nReward figures above were captured pre-v0.1.2, when `false_flag_weight` defaulted to 0.0\nand `must_not_say_weight` did not exist (hardcoded 0.0) — i.e. under a reward formula that\npaid zero attention to either penalty. Neither observed run had any false-flag or\nmust-not-say hits, so these particular numbers are unchanged under the current formula,\nbut they should not be read as validating the old weighting; re-run to get v0.1.2 numbers\non new models.\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly; `finding_recall`\nis the discriminative axis and is strict by construction -- credit requires engaging with\nthe specific planted issue, not mentioning its topic. `false_flag_rate > 0` and\n`must_not_say_hits > 0` both now subtract from reward (see How it's graded) --\n`false_flag_rate > 0` indicates the agent escalates noise, `must_not_say_hits > 0`\nindicates it tripped a planted legal-safety trap. `transcript_recall > finding_recall`\nindicates findings that never made it into the saved deliverable (a workflow-discipline\nfailure, not a knowledge one). Rollout-to-rollout variance is real at this scale; the\ndefault 3 rollouts per task are a minimum for stable comparisons.\n\n## Provenance\n\nTask content and answer keys are original work from a validated synthetic-task factory\n(35 tasks across 7 practice areas; this package ships the corporate governance pocket).\nAnchor compilation lineage is recorded in each task's `anchors.json` provenance block and\nin the stamped module header. This build has not been frozen; freeze SHA completion is a\nseparate manual step per SPEC section 6. See `NOTICE` for attribution details.\n\nThis is one environment in a per-practice-area family; contracts, corporate M&A, and\ncybersecurity & privacy pockets follow the same engine and scoring contract.\n\n## Known issues\n- **`finding_recall` is pure keyword/phrase-proximity matching, not semantic-content grading.**\n  A mechanical \"surgical shotgun\" — each finding's own published `core_groups`/`evidence`\n  vocabulary concatenated as word-salad, with zero legal reasoning, zero citations, zero fact\n  application, and `must_not_say`/distractor trigger words deliberately omitted — scores\n  **0.9143, statistically identical to a hand-crafted honest memo (0.9143)**. The vocabulary\n  needed to build this shotgun ships inside the installed package's `anchors.json`, so this is\n  not a hidden held-out property.\n- **`must_not_say` is trivially evaded.** The negation check only scans for any negation word\n  anywhere in the same 240-char window, not whether it grammatically negates the claim —\n  appending an unrelated \"...and this is not a close call at all\" after the forbidden statement\n  fully neutralizes the penalty (0.4000 either way).\n- **No cross-task identity check.** Content written for one matter transfers 28-43% finding_recall\n  credit onto unrelated matters for free.\n- **Sign-convention warning:** despite the \"same engine and scoring contract\" line above,\n  `employment-law-legal-agent-eval`'s `false_flag_weight`/`must_not_say_weight` are positive\n  magnitudes subtracted internally — the opposite convention from this env's weights (negative\n  numbers passed directly to the rubric). Do not reuse weight kwargs across siblings; doing so\n  silently inverts a penalty into a reward.\n- v0.1.2 fixes the wiring (penalties apply by default) but does not close the shotgun/evasion/\n  identity gaps above. Do not train unsupervised against this reward as evidence of genuine\n  legal-analysis quality — use only for coarse benchmarking of non-adversarial models.\n\n## Changelog\n\n- **v0.3.0** — **Negative-control passthrough fix (family stop-ship 2026-07-22).**\n  Negative-control findings (`kind=\"negative_control\"`, e.g. `ISS-007`) were lexically\n  earnable in `finding_recall`: a memo that merely uttered the planted \"negative control\"\n  jargon plus a couple of distractor evidence tokens was CREDITED, banking +1/n_findings of\n  `finding_recall` (measured **≈+0.086** net default reward at the 0.6 `finding_recall`\n  weight — `false_flag`/`must_not_say` do not offset it). In gated mode that NC credit sat\n  in the gate's `passthrough` set (the judge is never asked about NCs), so it was\n  un-removable. `finding_recall_details` now NEVER credits an NC: the NC stays in the\n  denominator (max recall = n_substantive / n_findings) and its lexical match is recorded as\n  telemetry only (`nc_matched_no_credit`, `nc_lexical_match_count`, `substantive_max_recall`).\n  `all_pass` now fires at `substantive_max_recall` rather than a literal `1.0` (every matter\n  ships exactly one NC, so the old `==1.0` gate was otherwise unreachable and, pre-fix, could\n  only be satisfied by tripping the NC trap). **This changes r-mode scoring: pre-v0.3.0\n  scores are comparable only for rollouts where no NC was lexically credited. Honest, NC-free\n  memos are byte-identical; any rollout that had banked NC credit now scores 1/n_findings\n  lower on `finding_recall`.** Mirrors the validated `agent-liability-gym` NC treatment.\n  Regression tests: `tests/test_nc_passthrough.py`.\n- **v0.2.1** — Doc-only. Replaced the pattern-ported placeholder validation status on the\n  `finding_recall` judge gate with the 2026-07-22 live judge-gate validation result. No\n  code changes.\n- **v0.2.0** — Added an optional, off-by-default **judge GATE** on `finding_recall`,\n  ported from the `agent-liability-gym` engine (AK judge-gate directive 2026-07-22). When\n  a `judge_model` is passed the judge re-adjudicates the *substantive* findings the keyword\n  scorer lexically credited and can only **remove** credit it cannot confirm is a genuine,\n  matter-specific treatment — it is never an additive reward term (an additive judge term\n  is itself farmable). Each confirmation is verified deterministically against a verbatim\n  memo quote (hallucinated evidence is discarded), uses a majority vote (`judge_votes`,\n  default 3, temperature 0), is cached in-process for repeatability, and fails loud\n  (`judge_error=1.0`; under `judge_strict` the rollout's finding credit is withheld rather\n  than silently scored ungated). New kwargs on `load_environment`/`build_environment`:\n  `judge_gate`, `judge_strict`, `judge_votes`, `judge_max_retries` (plus the existing\n  `judge_base_url`/`judge_api_key_var`). This **replaces** the previous weight-0 judge\n  *telemetry* rubric as the default when a `judge_model` is supplied; pass `judge_gate=False`\n  to restore that legacy telemetry rubric. **Default (no `judge_model`) scoring is\n  byte-identical to v0.1.2** — reward and every graded metric are unchanged, and no gate\n  telemetry metric is added in that mode (verified against the v0.1.2 build across all five\n  matters). Validation status: **Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini\n  policy, n=10 rollouts): judge_gated 10/10, judge_error 0/10, gate strictly remove-only,\n  reward algebra exact; removals spot-verified as genuine lexical false-positives. See\n  outputs/evals/ for the run.**\n  (the reference protocol validated 0 FP / 0 FN on agent-liability-gym with gate-v4 +\n  votes=3). Note this partially closes the \"surgical shotgun\" hole below *only when a\n  `judge_model` is supplied* — the deterministic default reward still has that gap.\n- **v0.1.2** — Fixed a reward-ignores-safety-criterion defect: `must_not_say_hits` (a\n  memo saying something a finding explicitly forbids, e.g. characterizing a protected\n  disclosure as something to \"sanitize\") was computed but summed into the reward with a\n  hardcoded 0.0 weight and no way to configure it, so tripping a legal-safety trap never\n  cost any reward. `must_not_say_hits` is now a normalized 0..1 rate (hits / applicable\n  must-not-say specs for the task, was previously a raw, unnormalized count) and is\n  wired to a new `must_not_say_weight` kwarg on `load_environment`/`build_environment`,\n  default **-0.10**. Also changed `false_flag_weight`'s default from `0.0` to **-0.15**\n  so distractor-escalation is penalized out of the box instead of only when a caller\n  opts in. Composite reward can now go below 0.0 for a memo that is both\n  incomplete/wrong and trips a penalty (intended — see How it's graded). Baseline\n  results table above predates this change.\n","encoding":"utf-8","truncated":false,"total_bytes":19886},"status":null}