{"data":{"kind":"file","path":"README.md","version_id":"o88l75m2o7bz3q0k60pso5zx","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8984,"modified_at":"2026-07-11T21:48:34.573000","content_hash":"d3a45c3fb1ea1f632a54fe82ac21126792f86470aa8847f1cd6eac58b97c327c"},"entries":[],"content":"# employment-law-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated employment counseling work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Six original, fully synthetic matters (no real client facts anywhere):\na nationwide noncompete rollout reaching $58K employees while the FTC rule's status is unsettled; an overtime exemption reclassification that fails the duties test; a pregnant-workers accommodation matter; a plant closure with WARN notice exposure; and a contractor classification review. Each matter contains planted issues the memo should raise — and\ndeliberately irrelevant documents a disciplined reviewer should decline to escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. There is\nno AI judging AI — the grading is deterministic, published in compiled form with the\npackage, and frozen by cryptographic hash so results are reproducible and comparable.\n\n**What the scores tell you.** Three things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise?* (distractor telemetry). Saving a file cannot compensate for weak legal work. Quoted\nsource text earns nothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\nemployment counseling work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine.\n\n## Quickstart\n\n```bash\nprime env install employment-law-legal-agent-eval\nprime eval run employment-law-legal-agent-eval -m <your-model>      # 6 tasks x 3 rollouts\nprime eval run employment-law-legal-agent-eval -m <your-model> -a '{\"task_id\":\"atlas-noncompete-001\"}'\n```\n\nThe agent gets four tools -- `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` -- and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward; judges are telemetry. The anchor specifications are\n**compiled from private answer keys** (never hand-written per task) and are designed to be\nfrozen by SHA-256 before any reported run. Matching is deliberately conservative --\nquoted source text is masked before scoring so pasting documents earns nothing, and a\nfinding only credits when the issue is named in its own terms *and* corroborated by\nseverity, remediation, or record-grounding evidence in the same context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nSix synthetic employment law matters, each a realistic dataroom-review delegation:\nterse partner instruction, 9 mixed-format documents (docx / pdf / xlsx / eml / txt), one\ndeclared `.docx` memo deliverable, 7 planted issues, and a 42-criterion rubric. All\ncontent is original synthetic material -- no third-party benchmark data, no real client\nfacts. Document corpora are intentionally compact in v0.1; corpus enrichment (full-length\nemployment policies, notices, personnel materials, multi-deliverable tasks) is the named\npath to v1.0.\n\n| Task | Title | Deliverable | Instruction |\n| --- | --- | --- | --- |\n| `atlas-noncompete-001` | Review AtlasBio Noncompete Rollout — Risk Memo | `noncompete-risk-memo.docx` | Review the attached documents against the compliance reference and prepare an employment risk memo. Output: noncompete-risk-memo.docx. |\n| `brightline-overtime-002` | Analyze BrightLine Manager Classification — Wage Memo | `overtime-classification-memo.docx` | Review the attached documents against the compliance reference and prepare a wage-hour memo. Output: overtime-classification-memo.docx. |\n| `cobalt-pwfa-003` | Review CobaltCare Pregnancy Accommodation — Accommodation Memo | `pwfa-accommodation-memo.docx` | Review the attached documents against the compliance reference and prepare a PWFA accommodation memo. Output: pwfa-accommodation-memo.docx. |\n| `delta-warn-004` | Assess DeltaWorks Reduction in Force — WARN Memo | `warn-analysis-memo.docx` | Review the attached documents against the compliance reference and prepare a WARN analysis memo. Output: warn-analysis-memo.docx. |\n| `ember-contractor-005` | Review EmberFleet Driver Classification — Classification Memo | `contractor-classification-memo.docx` | Review the attached documents against the compliance reference and prepare a contractor classification memo. Output: contractor-classification-memo.docx. |\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `source` | `\"factory\"` | `\"factory\"` loads the bundled anchored tasks; `\"lab\"` loads LAB tasks from an explicit `tasks_root` |\n| `task_id` | `\"all\"` | All five matters, or a single slug such as `atlas-noncompete-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `0.1` | Subtracted weight of the distractor-escalation penalty |\n| `must_not_say_weight` | `0.1` | Subtracted weight when a prohibited claim is detected |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | Optional weight-0 judge telemetry; requires `JUDGE_API_KEY` (or `judge_api_key_var`) only when set |\n| `tasks_root` | `None` | In factory mode, override the bundled task directory with the same layout; in LAB mode, required and must point at your own LAB checkout |\n\nNo environment variables are required by default.\n\n## LAB mode\n\n`source=\"lab\"` measures workflow discipline: did the agent persist every declared\ndeliverable under its exact filename with substantive content. It does not measure issue\ncoverage because LAB tasks do not ship with a compiled answer key, so there is no\n`finding_recall`, `false_flag_rate`, or similar anchored metric in this mode. Every LAB\nmetric carries a `lab_` prefix so LAB workflow telemetry is never confused with or\naveraged alongside factory metrics.\n\nLAB mode requires `tasks_root`; no default or bundled LAB tasks ship in the package. The\nloader discovers `tasks_root/<category>/<task-id>/task.json`, with source files in the\nsibling `documents/` directory. Each `task.json` must contain `instructions` as a string\nand `deliverables` as a dict whose keys are the declared filenames, for example\n`{\"memo.docx\": \"memo.docx\"}`. The internal row slug uses `category/task-id` to keep tasks\nfrom different categories distinct.\n\n## Baseline results\n\n| gpt-5.5 | 5 tasks × 3 rollouts | 0.594 | 0.324 | 15/15 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.406 | 0.010 | 15/15 |\n\nThe starkest small-model floor in the family: the 30B saves every deliverable but recalls\nalmost nothing (1%) — employment-law planted issues (duties tests, salary-basis mechanics,\nNLRA stay-or-pay interactions) demand specificity that consolidated generic memos don't\nreach. Credited findings were hand-audited against the answer keys; zero over-credit.\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly. `finding_recall`\nis the discriminative axis. `false_flag_rate > 0` indicates the agent escalates noise.\nRollout-to-rollout variance is real at this scale; the default 3 rollouts per task are a\nminimum for stable comparisons.\n\n## Provenance\n\nTask content and answer keys are original work from a validated synthetic-task factory\n(35 tasks across 7 practice areas; this package ships the employment law pocket).\nAnchor compilation lineage is recorded in each task's `anchors.json` provenance block and\nin the stamped module header. This build has not been frozen; freeze SHA completion is a\nseparate manual step per SPEC section 6. See `NOTICE` for attribution details.\n\nThis is one environment in a per-practice-area family; contracts, corporate governance,\nmergers & acquisitions, cybersecurity & privacy, AI law, and crypto law pockets follow\nthe same engine and scoring contract.\n","encoding":"utf-8","truncated":false,"total_bytes":8984},"status":null}