{"data":{"kind":"file","path":"README.md","version_id":"ei45gid4egbh5zblnzby8ntj","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":22224,"modified_at":"2026-07-23T03:45:35.402000","content_hash":"4732cd41ab88ad89361664e3d2ecb96cf4b059aa1ff4114d96ec4bfce2761937"},"entries":[],"content":"# crypto-law-legal-agent-eval\n\n**What this is.** A rigorous, repeatable test of whether an AI agent can handle a\npiece of delegated digital-asset work the way a careful junior lawyer would. The agent receives\na realistic matter file — nine mixed-format documents (agreements, fact schedules, board\nminutes, chat exports, reference memos) — and a short partner instruction: review the\nrecord and deliver a risk memorandum under a specific filename.\n\n**The matters.** Five original, fully synthetic matters (no real client facts anywhere):\na DeFi front-end relaunch without sanctions controls; a DePIN network review; a stablecoin\nmatter; a crypto ETF with cold-wallet signing and in-kind transfer gaps; and a token\nacquisition whose treasury has mixer-exposed wallet history. Alongside them are seven pilot\nmatters: Analyze Nereid USD Stablecoin Issuer Readiness, with a strictly source-grounded\nstablecoin issuer readiness memorandum; Tidegate Exchange Sanctions Audit, with a\nsanctions-compliance audit memorandum for the Tidegate Exchange release review; and Analyze\nVela Orchard Token M&A Treasury Diligence, with a strictly source-grounded token M&A sanctions\nand treasury diligence memorandum; Hosted-Mining Program Review, with a launch-risk memo\naddressing securities, fraud, verification, and reporting issues; Prime-Broker Onboarding, with\na risk memo addressing registration, transparency, asset-protection, current-posture,\nentity-mapping, and pending-authority findings; Alderpath RIA Custody Review, with a risk\nmemorandum addressing custody, platform, redemption, and current-law issues; and Pineglass DAO\nWrapper Structuring, with a risk memorandum addressing entity, governance, control, and\nimplementation issues. Each matter contains seven planted issues the memo\nshould raise — and deliberately irrelevant documents a disciplined reviewer should decline\nto escalate.\n\n**How it's graded.** The way a supervising lawyer would grade it, made mechanical: each\nmatter's drafting attorney prepared a private answer key (the issues, their severity, the\nexpected remediation), and scoring checks the delivered memo against that key. There is\nno AI judging AI — the grading is deterministic, published in compiled form with the\npackage, and frozen by cryptographic hash so results are reproducible and comparable.\n\n**What the scores tell you.** Four things a hiring partner would ask about a junior's\nwork: *Did the work product actually get delivered as instructed?* (deliverable\ncompletion) — *Did it catch the issues that matter?* (finding recall, the discriminative\naxis) — *Did it escalate noise?* (distractor telemetry). Quoted source text earns\nnothing; credit requires the agent's own analysis.\n\n**What it can help with.** Benchmarking models and vendors before trusting them with\ndigital-asset work; tracking whether a new model version actually improved on legal\nreasoning rather than fluency; and research on agent scaffolding — the same tasks can be\nrun with different tooling around the same model. This pocket is one of a seven-practice-\narea family sharing a single grading engine, so results are comparable across areas.\n\n## Quickstart\n\n```bash\nprime env install crypto-law-legal-agent-eval\nprime eval run crypto-law-legal-agent-eval -m <your-model>      # 12 tasks x 3 rollouts\nprime eval run crypto-law-legal-agent-eval -m <your-model> -a '{\"task_id\":\"helix-defi-001\"}'\n```\n\nThe agent gets four tools -- `list_documents`, `read_document` (paginated),\n`grep_documents`, and `save_deliverable` -- and up to 30 turns. Content that only appears\nin chat does not count: deliverables must be saved under their declared names, and a\n`memo.md` never satisfies a declared `memo.docx`.\n\n## Why deterministic scoring\n\nExecutable validators are the reward. By default no model judges another model; an\noptional judge acts only as a remove-only gate over the deterministic credit (see the\nfinding_recall judge gate under Environment arguments), never as an additive term. The anchor specifications are\n**compiled from private answer keys** (never hand-written per task) and are designed to be\nfrozen by SHA-256 before any reported run. Matching is deliberately conservative --\nquoted source text is masked before scoring so pasting documents earns nothing, and a\nfinding only credits when the issue is named in its own terms *and* corroborated by\nseverity, remediation, or record-grounding evidence in the same context window.\n\n**Pre-registration note:** the compiled anchors ship with the package because the reward\nneeds them. Do not tune prompts, scaffolds, or models against them; treat them as a\nheld-out grader.\n\n## Tasks\n\nTwelve crypto law matters, each a realistic dataroom-review delegation:\nterse partner instruction, 9 mixed-format documents (docx / pdf / xlsx / eml / txt), one\ndeclared `.docx` memo deliverable, 7 planted issues, and a 42-criterion rubric. All\nparty and client facts are fictionalized -- no third-party benchmark data, no real client\nfacts. The seven added pilot matters are grounded in real 2025-26 public crypto enforcement\nactions and related OFAC/FinCEN/NYDFS/GENIUS Act material, distinguishing them from the\noriginal five fully synthetic factory matters. Document corpora are intentionally compact\nin v0.1; corpus enrichment (full-length\ntoken materials, compliance references, custody controls, multi-deliverable tasks) is\nthe named path to v1.0.\n\n| scenario_id | scenario_slug | title | deliverable | instruction |\n| --- | --- | --- | --- | --- |\n| crypto-helix-defi-001 | helix-defi-001 | Analyze HelixSwap Sanctions Controls — Risk Memo | defi-sanctions-risk-memo.docx | Review the attached documents against the compliance reference and prepare a DeFi sanctions risk memo. Output: defi-sanctions-risk-memo.docx. |\n| crypto-mesh-depin-002 | mesh-depin-002 | Review MeshNode DePIN Rewards — Risk Memo | depin-token-risk-memo.docx | Review the attached documents against the compliance reference and prepare a DePIN token risk memo. Output: depin-token-risk-memo.docx. |\n| crypto-harbor-stablecoin-003 | harbor-stablecoin-003 | Assess HarborUSD Reserve Disclosures — Risk Memo | stablecoin-risk-memo.docx | Review the attached documents against the compliance reference and prepare a stablecoin disclosure memo. Output: stablecoin-risk-memo.docx. |\n| crypto-copper-etf-004 | copper-etf-004 | Analyze CopperRock ETF Custody Controls — Risk Memo | etf-custody-risk-memo.docx | Review the attached documents against the compliance reference and prepare a crypto ETF custody memo. Output: etf-custody-risk-memo.docx. |\n| crypto-atlas-token-ma-005 | atlas-token-ma-005 | Review AtlasPay Token Acquisition Diligence — Risk Memo | token-ma-risk-memo.docx | Review the attached documents against the compliance reference and prepare a token acquisition risk memo. Output: token-ma-risk-memo.docx. |\n| crypto-stablecoin-issuer-readiness | stablecoin-issuer-readiness | Analyze Nereid USD Stablecoin Issuer Readiness | stablecoin-issuer-readiness-memo.docx | Review the attached NUSD materials and prepare a stablecoin issuer readiness memo grounded only in the record. Output: stablecoin-issuer-readiness-memo.docx. |\n| crypto-token-ma-treasury | token-ma-treasury | Analyze Vela Orchard Token M&A Treasury Diligence | token-ma-treasury-risk-memo.docx | Review the attached documents and prepare a token M&A treasury sanctions diligence memo for Meridian. Output: token-ma-treasury-risk-memo.docx. |\n| crypto-exchange-sanctions-audit | exchange-sanctions-audit | Tidegate Exchange Sanctions Audit | exchange-sanctions-audit-memo.docx | Review the Tidegate Exchange sanctions-compliance record and prepare a risk memo identifying material sanctions issues and negative controls. Output: exchange-sanctions-audit-memo.docx. |\n| crypto-hosted-mining-program-review | hosted-mining-program-review | Hosted-Mining Program Review | hosted-mining-program-review-memo.docx | Review the supplied hosted-mining program record and prepare a launch-risk memo identifying material securities, fraud, verification, and reporting issues. Output: hosted-mining-program-review-memo.docx. |\n| crypto-prime-broker-onboarding | prime-broker-onboarding | Prime-Broker Onboarding | prime-broker-onboarding-risk-memo.docx | Review the Aurora Vale prime-broker onboarding record and prepare a risk memo identifying material registration, transparency, asset-protection, current-posture, entity-mapping, and pending-authority findings. Output: prime-broker-onboarding-risk-memo.docx. |\n| crypto-ria-custody-review | ria-custody-review | Alderpath RIA Custody Review | ria-custody-review-memo.docx | Review the Alderpath RIA custody record and prepare a risk memorandum identifying material custody, platform, redemption, and current-law issues. Output: ria-custody-review-memo.docx. |\n| crypto-dao-wrapper-structuring | dao-wrapper-structuring | Pineglass DAO Wrapper Structuring | dao-wrapper-structuring-memo.docx | Review the Pineglass DAO wrapper and relaunch record and prepare a risk memorandum identifying the material entity, governance, control, and implementation issues. Output: dao-wrapper-structuring-memo.docx. |\n\n## Environment arguments\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `source` | `\"factory\"` | `\"factory\"` loads the bundled anchored tasks; `\"lab\"` loads LAB tasks from an explicit `tasks_root` |\n| `task_id` | `\"all\"` | All twelve matters, or a single slug such as `helix-defi-001` |\n| `max_turns` | `30` | Tool-use turn budget |\n| `page_chars` | `6000` | Page size returned by `read_document` |\n| `false_flag_weight` | `-0.15` | Weight of the distractor-escalation penalty (`false_flag_rate`, 0-1); negative so escalating noise costs reward |\n| `must_not_say_weight` | `-0.10` | Weight of the forbidden-content penalty (`must_not_say_rate`, a normalized 0-1 rate of hits over probes); negative so violations cost reward |\n| `min_deliverable_chars` | `200` | Minimum content length for a deliverable to count |\n| `judge_model` | `None` | Off by default. When set, activates the **finding_recall judge gate** (below); requires `JUDGE_API_KEY` (or `judge_api_key_var`). Default (unset) scoring is byte-identical to the pre-gate deterministic reward |\n| `judge_gate` | `True` | When a `judge_model` is set, run the judge as an AND-gate inside `finding_recall` (remove-only). `False` disables the gate even with a judge configured |\n| `judge_strict` | `True` | On judge failure (API/parse error after retries), withhold ALL finding credit for that rollout (`judge_error=1.0`) rather than silently scoring ungated. `False` keeps the lexical credits, still flagged via `judge_error` |\n| `judge_votes` | `3` | Majority-vote calls per gate decision (temperature=0, parallel). An issue is confirmed only when a majority of votes confirm it |\n| `judge_base_url` / `judge_api_key_var` | `None` / `\"JUDGE_API_KEY\"` | Judge client endpoint and the env var holding its API key |\n| `tasks_root` | `None` | In factory mode, override the bundled task directory with the same layout; in LAB mode, required and must point at your own LAB checkout |\n\n### finding_recall judge gate (optional)\n\nBy default the reward is fully deterministic — no model judges another model. When a\n`judge_model` is supplied, the judge is wired as a **gate**, never an additive term: it\nre-examines each finding the keyword matcher lexically credited and can only **remove**\ncredit, never add it (an additive judge term is itself farmable). A finding survives the\ngate only if the judge confirms a genuine, matter-specific treatment **and** returns a\nverbatim memo quote that the code then checks is a real (normalized) substring of the memo\n— so a judge that hallucinates evidence loses that slot. Negative-control distractors are\nnever offered to the judge and are dropped from credit when the gate is active. Decisions\nuse majority vote across `judge_votes` temperature-0 calls with an in-process cache, so\ngated rewards are repeatable within a run. `judge_gated`, `judge_error`, and\n`finding_recall_lexical` (pre-gate recall) are exposed as weight-0 telemetry. This gate\nreplaces the former weight-0 `judge_coverage_estimate` monitor. Validation status:\npattern-ported from `agent-liability-gym` (0 FP / 0 FN protocol). Live-validated\n2026-07-22 (judge grok-4.5, gpt-5.4-mini policy, n=10 rollouts): judge_gated 10/10,\njudge_error 0/10, gate strictly remove-only, reward algebra exact; removals\nspot-verified as genuine lexical false-positives. See outputs/evals/ for the run.\n\nNo environment variables are required by default.\n\n## LAB mode\n\n`source=\"lab\"` measures workflow discipline: did the agent persist every declared\ndeliverable under its exact filename with substantive content. It does not measure issue\ncoverage because LAB tasks do not ship with a compiled answer key, so there is no\n`finding_recall`, `false_flag_rate`, or similar anchored metric in this mode. Every LAB\nmetric carries a `lab_` prefix so LAB workflow telemetry is never confused with or\naveraged alongside factory metrics.\n\nLAB mode requires `tasks_root`; no default or bundled LAB tasks ship in the package. The\nloader discovers `tasks_root/<category>/<task-id>/task.json`, with source files in the\nsibling `documents/` directory. Each `task.json` must contain `instructions` as a string\nand `deliverables` as a dict whose keys are the declared filenames, for example\n`{\"memo.docx\": \"memo.docx\"}`. The internal row slug uses `category/task-id` to keep tasks\nfrom different categories distinct.\n\n## Baseline results\n\n### v0.3.0 (12 tasks — current)\n\n> **Staleness caveat (v0.3.3):** the numbers below were captured under the pre-v0.3.3 formula\n> (false-flag and must-not-say weights were 0.0). New runs score lower whenever those penalties\n> fire and are not directly comparable to this table.\n\n| Model | Setup | Reward | Finding recall | Deliverables saved |\n| --- | --- | ---: | ---: | ---: |\n| gpt-5.5 | 12 tasks × 3 rollouts | 0.719 | 0.532 | 36/36 |\n| qwen3-30b-a3b-instruct-2507 | 12 tasks × 3 rollouts | 0.538 | 0.286 | 33/36 |\n\nThe frontier model leads on every one of the twelve matters. Its best single-memo\nrecall was 0.857 (twice: the exchange sanctions audit and the stablecoin readiness\nmatter). The 30B's three missing deliverables were provider-side request failures,\nscored as zero per the grading contract. Credited findings were hand-audited against\nthe answer keys at the criterion bar across every calibration round and sampled from\nthe baseline runs themselves; zero over-credit.\n\n### v0.1.x (original 5-task set — historical)\n\n| Model | Setup | Reward | Finding recall |\n| --- | --- | ---: | ---: |\n| gpt-5.5 | 5 tasks × 3 rollouts | 0.640 | 0.400 |\n| qwen3-30b-a3b-instruct-2507 | 5 tasks × 3 rollouts | 0.509 | 0.181 |\n\nHistorical numbers are not comparable across package versions (task sets differ).\n\n## Interpreting scores\n\nA competent agent should approach `deliverable_complete = 1.0` quickly; `finding_recall`\nis the discriminative axis. It is a lexical anchor matcher: credit requires the finding's\ncore and evidence vocabulary in proximity, which honest engagement produces — but see\nKnown Issues: keyword-density payloads can also satisfy it. `false_flag_rate > 0` indicates the\nagent escalates noise, and now costs reward directly (`false_flag_weight`, default\n`-0.15`). `must_not_say_rate > 0` indicates the memo leaned on content the system prompt\nexplicitly forbids (outside law, unstated facts, unreviewed documents); it is the\n`must_not_say_hits` raw count normalized to a 0-1 rate over the task's forbidden-content\nprobes, and also costs reward directly (`must_not_say_weight`, default `-0.10`).\n`transcript_recall > finding_recall` indicates findings that never made it into the saved\ndeliverable (a workflow-discipline failure, not a knowledge one). Rollout-to-rollout\nvariance is real at this scale; the default 3 rollouts per task are a minimum for stable\ncomparisons.\n\n## Known Issues\n- **Keyword-density shotgun reaches/exceeds honest reward.** A payload that stuffs each\n  finding's own evidence vocabulary (zero legal reasoning, zero document reads) scores 1.0000\n  vs 0.5714 for an honest hand-written baseline on the same task, and a single static 10.5KB\n  task-agnostic blob copy-pasted across all 12 tasks averages 0.9619 — above the README's own\n  gpt-5.5 honest baseline (0.719). `finding_recall` has no coherence/grounding check.\n- **`must_not_say` is evadable** via an unrelated negation word anywhere in the same scoring\n  window (one decoy \"not\" neutralizes the penalty).\n- Do not train unsupervised against this reward without a semantic/grounding gate. The\n  optional `judge_model` finding_recall gate (v0.4.0) is exactly that gate — it drops\n  lexically-credited findings the judge cannot confirm against a verbatim memo quote — off\n  by default. Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini policy, n=10\n  rollouts): judge_gated 10/10, judge_error 0/10, gate strictly remove-only, reward algebra\n  exact; removals spot-verified as genuine lexical false-positives. See outputs/evals/ for\n  the run.\n\n## Changelog\n\n- **v0.5.0** -- **Negative-control passthrough fix (family stop-ship 2026-07-22).**\n  Negative-control findings (`kind=\"negative_control\"`, e.g. `ISS-007`) were lexically\n  earnable in `finding_recall`: a memo that merely uttered the planted \"negative control\"\n  jargon plus a couple of distractor evidence tokens was CREDITED, banking +1/n_findings of\n  `finding_recall` (measured **≈+0.086** net default reward at the 0.6 `finding_recall`\n  weight — `false_flag`/`must_not_say` do not offset it). In gated mode that NC credit was\n  never re-adjudicated (the judge is never asked about NCs), so it was un-removable.\n  `finding_recall_details` now NEVER credits an NC: the NC stays in the denominator (max\n  recall = n_substantive / n_findings) and its lexical match is recorded as telemetry only\n  (`nc_matched_no_credit`, `nc_lexical_match_count`, `substantive_max_recall`). `all_pass`\n  now fires at `substantive_max_recall` rather than a literal `1.0` (every matter ships\n  exactly one NC, so the old `==1.0` gate was otherwise unreachable and, pre-fix, could only\n  be satisfied by tripping the NC trap). **This changes r-mode scoring: pre-v0.5.0 scores\n  are comparable only for rollouts where no NC was lexically credited. Honest, NC-free memos\n  are byte-identical; any rollout that had banked NC credit now scores 1/n_findings lower on\n  `finding_recall`.** Mirrors the validated `agent-liability-gym` NC treatment. Regression\n  tests: `tests/test_nc_passthrough.py`.\n- **v0.4.1** -- Doc-only. Replaced the pattern-ported placeholder validation status on the\n  `finding_recall` judge gate with the 2026-07-22 live judge-gate validation result. No\n  code changes.\n- **v0.4.0** -- Optional **finding_recall judge gate**. When a `judge_model` is passed, the\n  judge is now an AND-gate inside `finding_recall` that can only REMOVE lexically-credited\n  findings, never add credit; each surviving finding must carry a judge-supplied verbatim\n  memo quote that is verified mechanically as a real substring of the memo (hallucinated\n  evidence loses the slot). Decisions use `judge_votes` (default 3) majority voting at\n  temperature 0 with an in-process cache; on judge failure the gate is fail-loud\n  (`judge_error=1.0`) and, under `judge_strict` (default), withholds finding credit for the\n  rollout. New params: `judge_gate`, `judge_strict`, `judge_votes`, `judge_max_retries`.\n  New weight-0 telemetry: `judge_gated`, `judge_error`, `finding_recall_lexical` (pre-gate\n  recall). This **replaces the former weight-0 `judge_coverage_estimate` monitor**. Default\n  (no `judge_model`) scoring is byte-identical to v0.3.3. Validation: pattern-ported from\n  `agent-liability-gym` (0 FP on 19 exploit findings / 0 FN on 30 honest in that env).\n  Live-validated 2026-07-22 (judge grok-4.5, gpt-5.4-mini policy, n=10 rollouts):\n  judge_gated 10/10, judge_error 0/10, gate strictly remove-only, reward algebra exact;\n  removals spot-verified as genuine lexical false-positives. See outputs/evals/ for the\n  run.\n- **v0.3.3** -- Reward-wiring fix. `false_flag_rate` (distractor-escalation) and the new\n  `must_not_say_rate` (forbidden-content, normalized from the previous unbounded\n  `must_not_say_hits` raw count) now carry non-zero default weights (`-0.15` and `-0.10`\n  respectively) in the scalar reward. Previously both defaulted to weight `0.0` --\n  `false_flag_weight` was a caller-overridable kwarg nobody overrode, and\n  `must_not_say_hits` was wired at a literal `0.0` in the rubric's weights list -- so a\n  memo could hallucinate unsupported findings or say what the system prompt explicitly\n  forbade with zero reward consequence, scoring byte-identical to a clean run with matched\n  `deliverable_complete`/`finding_recall`. `must_not_say_hits` (the raw integer count) is\n  still reported for backward compatibility; `must_not_say_rate` is the new normalized\n  (0-1) metric the weight applies to.\n\n## Provenance\n\nThe original five tasks and answer keys are original work from a validated synthetic-task\nfactory (35 tasks across 7 practice areas; this package ships its crypto law pocket). The\nseven pilot tasks were compiled by the same `compile_anchors.py` mechanism from the separate\nbranch-side `pilot/crypto/` pipeline: three Lot-1 matters (stablecoin-issuer-readiness,\ntoken-ma-treasury, and exchange-sanctions-audit) and four Lot-2 matters\n(hosted-mining-program-review, prime-broker-onboarding, ria-custody-review, and\ndao-wrapper-structuring). They are grounded in real 2025-26 public enforcement actions with\nfictionalized parties and are not part of the factory's 35-task synthetic corpus.\nAnchor compilation lineage is recorded in each task's `anchors.json` provenance block and\nin the stamped module header. This build has not been frozen; freeze SHA completion is a\nseparate manual step per SPEC section 6. See `NOTICE` for attribution details.\n\nThis is one environment in a per-practice-area family; contracts, corporate governance,\nmergers & acquisitions, cybersecurity & privacy, employment law, and AI law pockets\nfollow the same engine and scoring contract.\n","encoding":"utf-8","truncated":false,"total_bytes":22224},"status":null}