{"data":{"kind":"file","path":"README.md","version_id":"cbhimbn5iwot7hfnk2x8qwfz","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":22735,"modified_at":"2026-07-23T03:58:59.965000","content_hash":"593c6bfbd0b4a40f2f646a74cefe000e2190d388ed9061d68f785ddccf8d8a8b"},"entries":[],"content":"# employment-law-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated employment counseling work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Six original, fully synthetic matters (no real client facts anywhere):\na nationwide noncompete rollout reaching $58K employees while the FTC rule's status is unsettled; an overtime exemption reclassification that fails the duties test; a pregnant-workers accommodation matter; a plant closure with WARN notice exposure; and a contractor classification review. Each matter contains planted issues the memo should raise — and\ndeliberately irrelevant documents a disciplined reviewer should decline to escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. There is\nno AI judging AI — the grading is deterministic, published in compiled form with the\npackage, and frozen by cryptographic hash so results are reproducible and comparable.\n\n**What the scores tell you.** Three things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise?* (distractor telemetry). Saving a file cannot compensate for weak legal work. Quoted\nsource text earns nothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\nemployment counseling work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine.\n\n## Quickstart\n\n```bash\nprime env install employment-law-legal-agent-eval\nprime eval run employment-law-legal-agent-eval -m <your-model>      # 6 tasks x 3 rollouts\nprime eval run employment-law-legal-agent-eval -m <your-model> -a '{\"task_id\":\"atlas-noncompete-001\"}'\n```\n\nThe agent gets four tools -- `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` -- and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward; judges are telemetry. The anchor specifications are\n**compiled from private answer keys** (never hand-written per task) and are designed to be\nfrozen by SHA-256 before any reported run. Matching is deliberately conservative --\nquoted source text is masked before scoring so pasting documents earns nothing, and a\nfinding only credits when the issue is named in its own terms *and* corroborated by\nseverity, remediation, or record-grounding evidence in the same context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nSix synthetic employment law matters, each a realistic dataroom-review delegation:\nterse partner instruction, 9 mixed-format documents (docx / pdf / xlsx / eml / txt), one\ndeclared `.docx` memo deliverable, 7 planted issues, and a 42-criterion rubric. All\ncontent is original synthetic material -- no third-party benchmark data, no real client\nfacts. Document corpora are intentionally compact in v0.1; corpus enrichment (full-length\nemployment policies, notices, personnel materials, multi-deliverable tasks) is the named\npath to v1.0.\n\n| Task | Title | Deliverable | Instruction |\n| --- | --- | --- | --- |\n| `atlas-noncompete-001` | Review AtlasBio Noncompete Rollout — Risk Memo | `noncompete-risk-memo.docx` | Review the attached documents against the compliance reference and prepare an employment risk memo. Output: noncompete-risk-memo.docx. |\n| `brightline-overtime-002` | Analyze BrightLine Manager Classification — Wage Memo | `overtime-classification-memo.docx` | Review the attached documents against the compliance reference and prepare a wage-hour memo. Output: overtime-classification-memo.docx. |\n| `cobalt-pwfa-003` | Review CobaltCare Pregnancy Accommodation — Accommodation Memo | `pwfa-accommodation-memo.docx` | Review the attached documents against the compliance reference and prepare a PWFA accommodation memo. Output: pwfa-accommodation-memo.docx. |\n| `delta-warn-004` | Assess DeltaWorks Reduction in Force — WARN Memo | `warn-analysis-memo.docx` | Review the attached documents against the compliance reference and prepare a WARN analysis memo. Output: warn-analysis-memo.docx. |\n| `ember-contractor-005` | Review EmberFleet Driver Classification — Classification Memo | `contractor-classification-memo.docx` | Review the attached documents against the compliance reference and prepare a contractor classification memo. Output: contractor-classification-memo.docx. |\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `source` | `\"factory\"` | `\"factory\"` loads the bundled anchored tasks; `\"lab\"` loads LAB tasks from an explicit `tasks_root` |\n| `task_id` | `\"all\"` | All five matters, or a single slug such as `atlas-noncompete-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `0.15` | Subtracted weight of the distractor-escalation penalty |\n| `must_not_say_weight` | `0.10` | Subtracted weight when a prohibited claim is detected |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | When set, activates the `finding_recall` **judge gate** (see below); requires `JUDGE_API_KEY` (or `judge_api_key_var`). When `None` (default) the gate is off and scoring is purely deterministic |\n| `judge_gate` | `True` | With a `judge_model`, run the judge as a remove-only gate on `finding_recall`. `False` restores a purely deterministic run even with a `judge_model` set |\n| `judge_strict` | `True` | On judge failure (API/parse error after retries), withhold ALL finding credit for the rollout and set `judge_error=1.0`. `False` keeps the lexical credits, still flagged |\n| `judge_votes` | `3` | Majority-vote calls per rollout (parallel, `temperature=0`); an issue is confirmed only if a majority of votes confirm it |\n| `judge_base_url` / `judge_api_key_var` | `None` / `\"JUDGE_API_KEY\"` | Judge client endpoint and env var holding its key |\n| `tasks_root` | `None` | In factory mode, override the bundled task directory with the same layout; in LAB mode, required and must point at your own LAB checkout |\n\nNo environment variables are required by default (the judge gate is off unless you pass a `judge_model`).\n\n## Judge gate (optional, off by default)\n\n`finding_recall` is a keyword/phrase-proximity matcher, so a \"surgical shotgun\" of a\nfinding's own published vocabulary can pass it without any legal reasoning (see Known\nissues). Passing a `judge_model` turns on a **gate** that hardens exactly that surface:\n\n- **Gate, never a bonus.** The judge only sees the findings the deterministic matcher\n  *already* credited, and can only *reject* them. It can never add credit the matcher\n  did not find (an additive judge term is itself farmable). Post-gate credit ⊆ lexical\n  credit, always.\n- **Mechanically verified confirmations.** For each finding it confirms, the judge must\n  return a verbatim `memo_quote`; the code checks that quote is a normalized substring of\n  the memo. A judge that hallucinates evidence loses that slot.\n- **Majority vote + cache.** `judge_votes` (default 3) parallel `temperature=0` calls with\n  a bare-call fallback; an issue survives only on a majority. Results are cached in-process\n  on (model, prompt version, votes, task, candidates, memo) so a run is repeatable.\n- **Fail-loud.** If a majority of votes error out, `judge_error=1.0` is set and (under the\n  default `judge_strict=True`) all finding credit for that rollout is withheld — never a\n  silent ungated score. A spike in `judge_error` across an eval is infra failure, not model\n  failure; do not read the accompanying rewards as capability signal.\n- **Injection-fenced prompt.** The memo is wrapped in a `<memo>…</memo>` fence and the\n  judge is told to ignore instructions inside it. The prompt is versioned\n  (`JUDGE_GATE_PROMPT_VERSION`).\n- **Telemetry (all weight 0):** `judge_gated`, `judge_error`, `finding_recall_lexical`\n  (pre-gate recall) and `legacy_issue_recall_lexical`, exposed alongside the post-gate\n  `finding_recall` / `legacy_issue_recall`.\n\n**Where the gate bites.** For the five recall-scored matters, `substantive_base` (hence the\nreward) equals `finding_recall`, so the gate lowers the reward directly. For\n`california-retaliation-006`, `substantive_base` is driven by the atomic **C-NNN** rubric,\nnot the issue-level credited set, so there the gate lowers only `legacy_issue_recall`\ntelemetry — not the reward. This is intentional: the atomic rubric is a separate, stronger\ndeterministic mechanism the gate does not need to second-guess.\n\n**Validation status:** pattern-ported from the validated `agent-liability-gym` gate\n(`gate-v4`, votes=3; 0 false-positives on 19 exploit findings, 0 false-negatives on 30\nhonest memos there). **Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini policy,\nn=10 rollouts): judge_gated 10/10, judge_error 0/10, gate strictly remove-only, reward\nalgebra exact; removals spot-verified as genuine lexical false-positives. See\noutputs/evals/ for the run.**\n\n## LAB mode\n\n`source=\"lab\"` measures workflow discipline: did the agent persist every declared\ndeliverable under its exact filename with substantive content. It does not measure issue\ncoverage because LAB tasks do not ship with a compiled answer key, so there is no\n`finding_recall`, `false_flag_rate`, or similar anchored metric in this mode. Every LAB\nmetric carries a `lab_` prefix so LAB workflow telemetry is never confused with or\naveraged alongside factory metrics.\n\nLAB mode requires `tasks_root`; no default or bundled LAB tasks ship in the package. The\nloader discovers `tasks_root/<category>/<task-id>/task.json`, with source files in the\nsibling `documents/` directory. Each `task.json` must contain `instructions` as a string\nand `deliverables` as a dict whose keys are the declared filenames, for example\n`{\"memo.docx\": \"memo.docx\"}`. The internal row slug uses `category/task-id` to keep tasks\nfrom different categories distinct.\n\n## Baseline results\n\n| gpt-5.5 | 5 tasks × 3 rollouts | 0.594 | 0.324 | 15/15 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.406 | 0.010 | 15/15 |\n\nThe starkest small-model floor in the family: the 30B saves every deliverable but recalls\nalmost nothing (1%) — employment-law planted issues (duties tests, salary-basis mechanics,\nNLRA stay-or-pay interactions) demand specificity that consolidated generic memos don't\nreach. Credited findings were hand-audited against the answer keys; zero over-credit.\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly. `finding_recall`\nis the discriminative axis. `false_flag_rate > 0` indicates the agent escalates noise.\nRollout-to-rollout variance is real at this scale; the default 3 rollouts per task are a\nminimum for stable comparisons.\n\n## Provenance\n\nTask content and answer keys are original work from a validated synthetic-task factory\n(35 tasks across 7 practice areas; this package ships the employment law pocket).\nAnchor compilation lineage is recorded in each task's `anchors.json` provenance block and\nin the stamped module header. This build has not been frozen; freeze SHA completion is a\nseparate manual step per SPEC section 6. See `NOTICE` for attribution details.\n\nThis is one environment in a per-practice-area family; contracts, corporate governance,\nmergers & acquisitions, cybersecurity & privacy, AI law, and crypto law pockets follow\nthe same engine and scoring contract.\n\n## Known issues\n- **`finding_recall`/`substantive_base` is pure keyword/phrase-proximity matching, not semantic\n  grading.** A \"surgical shotgun\" built purely from each finding's own published vocabulary (no\n  legal reasoning, no citations) scores **1.0000 — statistically identical to a hand-crafted\n  honest memo (also 1.0000)** in the default deterministic-only mode. **Mitigation (v0.3.0):**\n  pass a `judge_model` to enable the remove-only `finding_recall` gate (see \"Judge gate\"), which\n  rejects vocabulary-without-claims candidates. Live-validated 2026-07-22 (judge grok-4.5,\n  gpt-5.4-mini policy, n=10 rollouts): judge_gated 10/10, judge_error 0/10, gate strictly\n  remove-only, reward algebra exact; removals spot-verified as genuine lexical\n  false-positives. See outputs/evals/ for the run.\n- ~~**CRITICAL: the floor clamp (`max(0.0, min(1.0, base - penalty))`) zeroes the false-flag/\n  must-not-say penalty whenever `substantive_base == 0`.**~~ **FIXED in v0.5.0 (C19).** The\n  floor is now `-(false_flag_weight + must_not_say_weight)` (= `-0.25` at the default\n  `0.15`/`0.10` weights), so a memo that hallucinates every distractor as real risk AND trips\n  every must-not-say safety trap now scores `-0.25` — strictly below a harmless, content-free\n  memo (`0.0`) — instead of the pre-fix byte-identical `0.0`. Reward range is now `[-0.25, 1]`\n  (was `[0, 1]`); the `min(1.0, ...)` ceiling is unchanged and honest, penalty-free memos are\n  byte-identical to prior versions. See the v0.5.0 changelog entry.\n- **`must_not_say_hits` is an unnormalized raw count**, capped at the same penalty for 1 hit as\n  for 5+ hits — unlike corporate-governance/crypto-law/ai-law, which normalize to a 0..1 rate.\n- **Sign-convention warning:** this env's `false_flag_weight`/`must_not_say_weight` are positive\n  magnitudes subtracted internally — the opposite convention from corporate-governance/crypto-law/\n  ai-law. Passing this env's documented \"family standard\" convention (`-0.15`/`-0.10`) into this\n  env's `substantive_reward` **rewards** false-flagging instead of penalizing it (verified: 0.75\n  dirty vs. 0.60 clean). Do not reuse weight kwargs across siblings.\n- v0.2.2 only raised a default weight; it closes none of the issues above. Do not train\n  unsupervised against this reward as evidence of genuine legal-analysis quality or safety-trap\n  avoidance.\n\n## Changelog\n\n**0.5.0** -- **Reward-floor fix so penalties always bite (C19, 2026-07-22).** The load-bearing\n`substantive_score` reward clamped with `max(0.0, min(1.0, base - penalty))`. Because\n`base = deliverable_complete * substantive_base` collapses to 0 whenever the memo recalls none\nof the planted findings, `base - penalty` was already negative there and the `max(0.0, ...)`\nfloor snapped it back to `0.0` — so the `false_flag`/`must_not_say` penalty **never fired at\nzero recall**, the exact case it exists to catch. A memo that hallucinated every distractor as\nreal risk AND tripped every must-not-say legal-safety trap scored **byte-identical (`0.0`) to a\nharmless, content-free memo**. The floor is now `-(false_flag_weight + must_not_say_weight)`;\nboth penalty terms are bounded in `[0, 1]`, so that sum is the maximum possible penalty and the\npenalty now always bites while the `min(1.0, ...)` ceiling is unchanged. **The reward range is\nnow `[-0.25, 1]`** (was `[0, 1]`) at the default `0.15`/`0.10` weights, where the floor is\nexactly `-0.25`. The reckless memo above now scores `-0.25`, strictly below the content-free\n`0.0`. Family-consistent with `contract-review-agent-v2` (documented `[-0.25, 1.0]` range) and\n`agent-liability-gym` (weighted-sum penalties bite whenever any positive term exists.)\n**Score comparability: honest, penalty-free memos are byte-identical to `0.4.0`; pre-0.5.0\nscores are comparable except rows that had a nonzero `false_flag`/`must_not_say` penalty at\nzero recall (previously floored to `0.0`, now negative down to `-0.25`).** Regression tests:\n`tests/test_c19_reward_floor.py` (red/green: hallucinate-everything strictly below content-free,\nhonest byte-identical, floor exactly `-0.25` at defaults); `test_penalties_subtract_and_reward_is_clamped`\nin `tests/test_california_retaliation_006.py` updated for the new floor.\n\n**0.4.0** -- **Negative-control passthrough fix (family stop-ship 2026-07-22).**\nNegative-control findings (`kind=\"negative_control\"`, e.g. `ISS-007` / `I-007`) were\nlexically earnable in `finding_recall` / `substantive_base`: uttering the planted \"negative\ncontrol\" jargon plus a couple of distractor tokens CREDITED the NC, inflating\n`substantive_base` by 1/n_findings and thus the load-bearing `substantive_score` reward by\nup to **+0.143** (reward = `deliverable_complete` × `substantive_base` − penalties;\n`false_flag`/`must_not_say` do not offset it). `finding_recall_details` now NEVER credits an\nNC: the NC stays in the denominator (max recall = n_substantive / n_findings) and its lexical\nmatch is kept as telemetry only (`nc_matched_no_credit`, `nc_lexical_match_count`,\n`substantive_max_recall`). `all_pass` now fires at `substantive_max_recall` on the\nrecall-scored matters (still a literal `1.0` on the atomic-rubric matter,\n`california-retaliation-006`, whose reward path is the C-NNN atomic criteria and is\nunaffected — its golden-answer coverage test was updated to stop expecting NC credit). **This\nchanges r-mode scoring: pre-0.4.0 scores are comparable only for rollouts where no NC was\nlexically credited; honest, NC-free memos are byte-identical.** Mirrors the validated\n`agent-liability-gym` NC treatment. Regression tests: `tests/test_nc_passthrough.py`.\n\n**0.3.1** -- Doc-only. Replaced the pattern-ported placeholder validation status on the\n`finding_recall` judge gate with the 2026-07-22 live judge-gate validation result. No\ncode changes.\n\n**0.3.0** -- Added an optional judge **GATE** on `finding_recall`. When `judge_model` is\npassed, each lexically-credited finding must be CONFIRMED by an LLM judge that returns a\nverbatim memo quote (checked mechanically as a normalized substring); unconfirmed findings\nare dropped before `finding_recall`/`substantive_base` -- and thus the reward -- are\ncomputed. The gate can only REMOVE credit the deterministic matcher granted, never add it\n(an additive judge term is farmable), so post-gate credit is always a subset of lexical\ncredit. Majority vote (`judge_votes`, default 3, parallel, `temperature=0` with a bare-call\nfallback); in-process cache keyed on (model, prompt version, votes, task, candidates, memo);\nfail-loud (`judge_error=1.0` and, under the default `judge_strict=True`, all finding credit\nwithheld on judge failure -- never a silent ungated score). The legacy weight-0 factory\n`JudgeRubric` \"coverage estimate\" telemetry was REMOVED and replaced by this gate. New env\nargs: `judge_gate` (default `True`), `judge_strict` (default `True`), `judge_votes` (default\n`3`), `judge_max_retries` (default `2`). New weight-0 telemetry metrics: `judge_gated`,\n`judge_error`, `finding_recall_lexical`, `legacy_issue_recall_lexical`. **OFF by default:**\nwith `judge_model=None` none of the above runs and r-mode scoring is byte-identical to\n`0.2.2` (verified: reward-path diff empty on GOLDEN; every pre-existing test passes\nunchanged). **Caveat:** for `california-retaliation-006` `substantive_base` is atomic-rubric\ndriven, so the gate lowers only that matter's `legacy_issue_recall` telemetry, not its\nreward; on the five recall-scored matters the gate lowers the reward directly.\n**Validation status: pattern-ported from the validated agent-liability-gym gate (`gate-v4`,\nvotes=3). Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini policy, n=10 rollouts):\njudge_gated 10/10, judge_error 0/10, gate strictly remove-only, reward algebra exact;\nremovals spot-verified as genuine lexical false-positives. See outputs/evals/ for the\nrun.**\n\n**0.2.2** -- Raised `false_flag_weight` default from `0.1` to `0.15` and confirmed\n`must_not_say_weight` at its already-correct `0.10`, matching this engine family's\nstandard penalty-weight convention (`-0.15`/`-0.10`, applied here as positive\nmagnitudes subtracted inside `substantive_reward`). This only changes the *default*\nused when an eval doesn't pass its own `false_flag_weight`/`must_not_say_weight`\nenv-args; the reward shape itself (`deliverable_complete * substantive_base`, minus\nthe false-flag/must-not-say penalty, clamped to `[0, 1]` -- no additive floor) was\nalready in place as of the prior `0.2.1` push and is unchanged by this release.\n\n**Version-history note (read before comparing scores across runs).** The declared\nversion `\"0.1.0\"` was pushed to the hub three separate times with materially\ndifferent reward code, so `\"0.1.0\"` alone does not identify a reproducible scoring\nfunction:\n- The first two `0.1.0` pushes (2026-07-08 and 2026-07-11 07:00 UTC) computed\n  `reward = 0.4*deliverable_complete + 0.6*finding_recall` -- a flat 40% credit for\n  merely saving a correctly-named, >=200-char deliverable, regardless of content\n  quality. **The Opus 4.8 eval `j0kwnb9w24ktug0gh42ckr74` was run under this\n  formula** and its scores are not comparable to any later run.\n- The third `0.1.0` push (2026-07-11 07:06 UTC) switched to the no-floor\n  multiplicative form (`deliverable_complete * finding_recall`, no additive credit\n  for an empty-content memo), which is the shape `0.2.0`/`0.2.1`/`0.2.2` all keep.\n- Every push from `0.2.1` onward additionally has `false_flag_weight`/\n  `must_not_say_weight` penalty terms wired into the reward (subtracted, not\n  floored); in every hosted eval sampled so far those penalties happened to be 0\n  (no false flags or prohibited claims fired), so observed reward equaled\n  `finding_recall` exactly in those runs -- that is a property of the sampled\n  completions, not evidence that the penalty terms are absent from the code.\n\nIf you need to compare scores across runs, record and compare the environment's\ncontent hash (`prime env version list`), not the semver tag alone -- `\"0.1.0\"` and\neven a single `\"0.2.1\"` tag (pushed twice, `7b97b17c` then `b495511a`) are not\nsufficient on their own.\n","encoding":"utf-8","truncated":false,"total_bytes":22735},"status":null}