{"data":{"kind":"file","path":"README.md","version_id":"g7u8ik9ng9pqfp1x73l3vnae","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9756,"modified_at":"2026-07-10T14:46:14.278000","content_hash":"a3f820fcd491d58dedfa52adc940dc7b50941a02e3cb0a94ff5b6a15d0d008a2"},"entries":[],"content":"# contracts-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated contract review work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Five original, fully synthetic matters (no real client facts anywhere):\na SaaS master services agreement whose data-processing addendum quietly overrides the liability cap; a component supply agreement with an inconspicuous warranty disclaimer and a failing exclusive remedy; a consumer subscription with auto-renewal disclosure and cancellation problems; a professional-services agreement with IP-indemnity and acceptance-criteria gaps; and a construction change-order claim with notice and lien-waiver defects. Each matter contains seven planted issues the memo should raise — and\ndeliberately irrelevant documents a disciplined reviewer should decline to escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. There is\nno AI judging AI — the grading is deterministic, published in compiled form with the\npackage, and frozen by cryptographic hash so results are reproducible and comparable.\n\n**What the scores tell you.** Three things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise?* (distractor telemetry). Quoted source text earns\nnothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\ncontract review work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine, so results are comparable across areas.\n\n## Quickstart\n\n```bash\nprime env install narcolepticchicken/contracts-legal-agent-eval\nprime eval run contracts-legal-agent-eval -m <your-model>      # 5 tasks × 3 rollouts\nprime eval run contracts-legal-agent-eval -m <your-model> -a '{\"task_id\":\"neon-saas-001\"}'\n```\n\nThe agent gets four tools — `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` — and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward; judges are telemetry. The anchor specifications are\n**compiled from private answer keys** (never hand-written per task), frozen by SHA-256\nbefore any reported run, and calibrated in both directions against real model memos:\ncredited findings were manually audited against the answer keys for over-credit\n(disqualifying by design), and non-credits audited for under-credit. Matching is\ndeliberately conservative — quoted source text is masked before scoring so pasting\ndocuments earns nothing, and a finding only credits when the issue is named in its own\nterms *and* corroborated by severity, remediation, or record-grounding evidence in the\nsame context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nFive synthetic contracts matters (SaaS MSA, supply agreement, consumer subscription,\nprofessional services, construction change-order), each a realistic dataroom-review\ndelegation: terse partner instruction and 9 mixed-format documents (docx / pdf / xlsx / eml /\ntxt). The primary operative agreement in each matter is 29,633–29,819 extracted characters,\nand the complete corpus is 76,379–76,670 extracted characters. Each task requires a declared\n`.docx` memo; `neon-saas-001` and `quartz-services-004` also require an\n`issues-table.xlsx`. All content is original synthetic material — no third-party benchmark\ndata, no real client facts.\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `source` | `\"factory\"` | `\"factory\"` loads the bundled anchored tasks; `\"lab\"` loads LAB tasks from an explicit `tasks_root` |\n| `task_id` | `\"all\"` | All five matters, or a single slug such as `neon-saas-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `0.0` | Weight of the distractor-escalation penalty |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | Optional weight-0 judge telemetry; requires `JUDGE_API_KEY` (or `judge_api_key_var`) only when set |\n| `tasks_root` | `None` | In factory mode, override the bundled task directory with the same layout; in LAB mode, required and must point at your own LAB checkout |\n\nNo environment variables are required by default.\n\n## LAB mode\n\n`source=\"lab\"` measures workflow discipline: did the agent persist every declared\ndeliverable under its exact filename with substantive content. It does not measure issue\ncoverage because LAB tasks do not ship with a compiled answer key, so there is no\n`finding_recall`, `false_flag_rate`, or similar anchored metric in this mode. Every LAB\nmetric carries a `lab_` prefix so LAB workflow telemetry is never confused with or\naveraged alongside factory metrics.\n\nLAB mode requires `tasks_root`; no default or bundled LAB tasks ship in the package. The\nloader discovers `tasks_root/<category>/<task-id>/task.json`, with source files in the\nsibling `documents/` directory. Each `task.json` must contain `instructions` as a string\nand `deliverables` as a dict whose keys are the declared filenames, for example\n`{\"memo.docx\": \"memo.docx\"}`. The internal row slug uses `category/task-id` to keep tasks\nfrom different categories distinct.\n\n## Baseline results\n\n### v0.3.0 (enriched corpus — current)\n\n| Model | Setup | Reward | Finding recall | Deliverables saved |\n| --- | --- | ---: | ---: | ---: |\n| gpt-5.5 | 5 tasks × 3 rollouts | 0.766 | 0.610 | 15/15 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.352 | 0.143 | 10/15 |\n\nThe enriched corpus widens frontier separation sharply (reward 2.2×, recall 4.3×). The\ngap is workflow discipline, not knowledge: on the two tasks that require a second\ndeliverable (an issues-table spreadsheet alongside the memo), the 30B failed to file\nanything in 5 of 6 rollouts — it spends its whole turn budget reading the ~30K-character\noperative agreement and never produces the work product. The frontier model filed every\ndeliverable and its recall *rose* relative to the short-document corpus. Credited\nfindings were manually audited at the criterion bar (zero over-credit; sampled audits\non every rollout batch).\n\n### v0.2.x (short-document corpus — historical)\n\n**Measured on the earlier thin-document corpus; NOT comparable to v0.3.0: the documents\nand, for two tasks, the deliverable requirements are materially different.**\n\n| Model | Setup | Reward | Finding recall | Deliverables saved | False flags |\n| --- | --- | ---: | ---: | ---: | ---: |\n| gpt-5.5 | 5 tasks × 3 rollouts | 0.640 | 0.400 | 15/15 | 0.0 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.514 | 0.190 | 15/15 | 0.0 |\n| qwen3-235b-a22b-instruct-2507 | 5 tasks × 3 rollouts | 0.429 | 0.181 | 12/15 | 0.0 |\n| qwen3.6-27b | 1 task × 1 rollout (indicative) | 0.657 | 0.429 | 1/1 | 0.0 |\n\nNotes from the validation runs: the frontier model separates cleanly and stably from the\n30B class across every rollout batch, at 40% recall — ample headroom in both directions.\nThe two Qwen MoE models are statistically tied at this scale (their per-batch ordering\nswaps, as expected for tied models; treat differences of this size as noise at n=5).\nThe 235B's lower reward is workflow discipline, not knowledge: it failed to persist its\ndeliverable in 3 of 15 rollouts. Credited findings from the frontier run were manually\naudited against the answer keys — no false credits.\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly; `finding_recall`\nis the discriminative axis and is strict by construction — credit requires engaging with\nthe specific planted issue, not mentioning its topic. `false_flag_rate > 0` indicates the\nagent escalates noise. `transcript_recall > finding_recall` indicates findings that never\nmade it into the saved deliverable (a workflow-discipline failure, not a knowledge one).\nRollout-to-rollout variance is real at this scale; the default 3 rollouts per task are a\nminimum for stable comparisons.\n\n## Provenance\n\nTask content and answer keys are original work from a validated synthetic-task factory\n(35 tasks across 7 practice areas; this package ships the contracts pocket). Anchor\ncompilation, freeze SHAs, and calibration lineage are recorded in each task's\n`anchors.json` provenance block and in the stamped module header. See `NOTICE` for\nattribution details.\n\nThis is the first environment in a per-practice-area family; corporate M&A, data privacy\n& cybersecurity, and corporate governance pockets follow the same engine and scoring\ncontract.\n","encoding":"utf-8","truncated":false,"total_bytes":9756},"status":null}