{"data":{"kind":"file","path":"README.md","version_id":"lr2m0jf0kwwt5jj1jn5p78sp","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8201,"modified_at":"2026-07-08T15:40:13.939000","content_hash":"34f6c7c8423d6cd11f71e4c7984ef6e9085e0bf39b48327864248b0660bb1239"},"entries":[],"content":"# mergers-acquisitions-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated M&A diligence work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Five original, fully synthetic matters (no real client facts anywhere):\nan HSR readiness assessment with a size-of-transaction calculation gap and undisclosed competitive overlaps; a merger risk memo where non-binding agency guidelines are treated as law; an earnout claim undermined by the buyer's own operational choices; an indemnity notice with defective timing; and a closing where the target breached its ordinary-course covenant. Each matter contains seven planted issues the memo should raise — and\ndeliberately irrelevant documents a disciplined reviewer should decline to escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. There is\nno AI judging AI — the grading is deterministic, published in compiled form with the\npackage, and frozen by cryptographic hash so results are reproducible and comparable.\n\n**What the scores tell you.** Three things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise?* (distractor telemetry). Quoted source text earns\nnothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\nM&A diligence work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine, so results are comparable across areas.\n\n## Quickstart\n\n```bash\nprime env install mergers-acquisitions-legal-agent-eval\nprime eval run mergers-acquisitions-legal-agent-eval -m <your-model>      # 5 tasks x 3 rollouts\nprime eval run mergers-acquisitions-legal-agent-eval -m <your-model> -a '{\"task_id\":\"apex-hsr-001\"}'\n```\n\nThe agent gets four tools — `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` — and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward; judges are telemetry. The anchor specifications are\n**compiled from private answer keys** (never hand-written per task), frozen by SHA-256\nbefore any reported run, and calibrated in both directions against real model memos:\ncredited findings were manually audited against the answer keys for over-credit\n(disqualifying by design), and non-credits audited for under-credit. Matching is\ndeliberately conservative — quoted source text is masked before scoring so pasting\ndocuments earns nothing, and a finding only credits when the issue is named in its own\nterms *and* corroborated by severity, remediation, or record-grounding evidence in the\nsame context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nFive synthetic mergers & acquisitions matters, each a realistic M&A dataroom-review\ndelegation: terse partner instruction, 9 mixed-format documents (docx / pdf / xlsx / eml /\ntxt), one declared `.docx` memo deliverable, 7 planted issues, and a 42-criterion rubric.\nAll content is original synthetic material — no third-party benchmark data, no real\nclient facts. Document corpora are intentionally compact in v0.1; corpus enrichment\n(full-length deal files, multi-deliverable tasks) is the named path to v1.0.\n\n| Task | Title | Deliverable | Instruction |\n| --- | --- | --- | --- |\n| `apex-hsr-001` | Review ApexMed HSR Filing Readiness — Closing Memo | `hsr-readiness-memo.docx` | Review the attached documents against the compliance reference and prepare an HSR readiness memo. |\n| `bluepeak-platform-002` | Assess BluePeak Platform Merger Risk — Antitrust Memo | `merger-risk-memo.docx` | Review the attached documents against the compliance reference and prepare an antitrust risk memo. |\n| `cedar-earnout-003` | Review CedarLabs Earnout Dispute — Claim Memo | `earnout-claim-memo.docx` | Review the attached documents against the compliance reference and prepare an earnout claim memo. |\n| `drift-indemnity-004` | Assess DriftWare Indemnity Notice — Claim Memo | `indemnity-notice-memo.docx` | Review the attached documents against the compliance reference and prepare an indemnity notice memo. |\n| `ember-ordinary-course-005` | Review EmberCloud Pre-Closing Covenant — Closing Memo | `closing-covenant-memo.docx` | Review the attached documents against the compliance reference and prepare a closing covenant memo. |\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `task_id` | `\"all\"` | All five matters, or a single slug such as `apex-hsr-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `0.0` | Weight of the distractor-escalation penalty |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | Optional weight-0 judge telemetry; requires `JUDGE_API_KEY` (or `judge_api_key_var`) only when set |\n| `tasks_root` | `None` | Override the bundled task directory (same layout) |\n\nNo environment variables are required by default.\n\n## Baseline results\n\n| Model | Setup | Reward | Finding recall | Deliverables saved | False flags |\n| --- | --- | ---: | ---: | ---: | ---: |\n| gpt-5.5 | 5 tasks × 3 rollouts | 0.589 | 0.314 | 15/15 | 0.0 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.451 | 0.086 | 15/15 | 0.0 |\n\nThe frontier model separates cleanly and stably from the 30B class in every rollout batch,\nat 31% recall — substantial headroom in both directions. M&A recall runs lower than the\ncontracts pocket for the same models: the planted issues (HSR mechanics, earnout covenants,\nclosing conditions) are more specialized, and weaker models tend to consolidate them into\ngeneric process findings that the criterion-faithful checker correctly declines to credit.\nCredited findings from both models' runs were manually audited against the answer keys —\nno false credits.\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly; `finding_recall`\nis the discriminative axis and is strict by construction — credit requires engaging with\nthe specific planted issue, not mentioning its topic. `false_flag_rate > 0` indicates the\nagent escalates noise. `transcript_recall > finding_recall` indicates findings that never\nmade it into the saved deliverable (a workflow-discipline failure, not a knowledge one).\nRollout-to-rollout variance is real at this scale; the default 3 rollouts per task are a\nminimum for stable comparisons.\n\n## Provenance\n\nTask content and answer keys are original work from a validated synthetic-task factory\n(35 tasks across 7 practice areas; this package ships the mergers & acquisitions pocket).\nAnchor compilation and calibration lineage are recorded in each task's `anchors.json`\nprovenance block and in the stamped module header. See `NOTICE` for attribution details.\n\nThis is one environment in a per-practice-area family; contracts, data privacy &\ncybersecurity, and corporate governance pockets follow the same engine and scoring\ncontract.\n","encoding":"utf-8","truncated":false,"total_bytes":8201},"status":null}