{"data":{"kind":"file","path":"README.md","version_id":"q0mcbbcr0xpe232xe0hmclcl","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":13092,"modified_at":"2026-07-23T13:29:38.143000","content_hash":"0de823f95f5aabd22265ce043e1506f6a5a2a6c4c45d92e631c54ad96bcab6d5"},"entries":[],"content":"# crypto-law-howey\n\nA leakage-resistant Verifiers v1 environment for training and evaluating small\nmodels on closed-universe U.S. crypto-offering issue spotting. Each scenario\nrequires factor-by-factor Howey analysis, exact quoted evidence, a public legal\nconnection, a benchmark conclusion, authority IDs, and targeted missing facts.\nIt is an educational benchmark, not legal advice.\n\nThe scope is deliberately narrow: a synthetic federal investment-contract triage\nrubric, not a comprehensive statement of U.S. crypto law as of the card date and\nnot a prediction of any court's result. The common-enterprise convention uses\nhorizontal pooling/linkage. District-court token cases are record-specific\nillustrations; the Supreme Court decisions supply the controlling closed-universe\nprinciples. Before any real-world deployment, qualified counsel should review the\nauthority summaries, operational rules, and representative outputs.\n\n## Contract\n\n- Task: one coherent scenario and one response containing exactly one structured JSON object.\n- Train/dev/test: 480/120/80 reserved coherent feature graphs with zero graph overlap. Atomic fact clauses, question templates, and entity vocabularies are split-owned.\n- Grounding: `factor_analyses` contains one object per factor with a disposition matching the top-level label, an exact 2–40 word fact-record quote, a factor-specific public connection enum, and one concise evidence-bound reasoning sentence. The default deterministic scoring aligns quotes to generator-owned factor spans and does not use an LLM judge. **Known issue (default reward)**: `factor_accuracy` does not verify semantic relevance of the cited quote to its factor, and a zero-reasoning keyword-pattern classifier reaches ~85% of max reward on held-out data — do not treat high default reward as evidence of legal reasoning (see Known Issues). v0.5.0 adds an **optional, off-by-default judge GATE** (see Judge gate) that targets exactly this gap.\n- Reward: deterministic weighted score over four factor labels (40%), conclusion (22%), missing-fact F1 (13%), per-factor grounding (10%), route consistency (7%), authority precision/recall (5%), and disclaimer (3%). Invalid schema is gated to zero.\n- Harness: every comparator receives the same closed-universe authority card and exact output schema. A public-contract route validator treats the model's exact evidence span and connection enum as its substantive choices and records a derived view (the model's stated factor labels/conclusion/missing-facts alongside the values implied by its own connections and the decision rule) plus any divergences in `harness_trace`. It never reads private gold, rejects nonexact quotes or unknown routes, and retains the raw response. Crucially it does not rewrite the response: every metric scores the model's raw final message, so a route inconsistency is penalized (`route_consistency` and the accuracy metrics) rather than silently repaired. The schema-only repair user remains available for a frozen two-turn protocol and contains no gold or reward details.\n\n## Judge gate (v0.5.0, off by default)\n\nPass `judge_model` to `load_environment` (optionally `judge_base_url`,\n`judge_api_key_var` [default `JUDGE_API_KEY`], `judge_gate`, `judge_strict`,\n`judge_votes` [default 3]) to gate the `factor_accuracy` credited set. For each\nfactor whose top-level label already matches the answer key, the judge must\nconfirm that the factor's cited `evidence_quote` genuinely PERTAINS to that Howey\nfactor; a factor the judge rejects loses its `1/4` of `factor_accuracy` (and hence\n`0.40 × 1/4` of `task_reward`). This directly targets the ~85% keyword-shortcut\nhole below: a correct label backed by evidence about the wrong factor no longer\nearns full factor credit.\n\nProperties (enforced in `crypto_law_howey/judge_gate.py`, covered by\n`tests/test_judge_gate.py`):\n\n- **Gate, never additive.** The judge can only REMOVE label-match credit, never\n  add it — a wrong label stays uncredited regardless of what the judge says.\n- **Deterministic confirmation.** Each confirmation carries a `memo`-style verbatim\n  span the code checks mechanically: the judge's returned quote must be a\n  normalized substring of that factor's own cited `evidence_quote`, so a\n  hallucinated confirmation is discarded.\n- **Majority vote + cache.** `judge_votes` parallel calls at `temperature=0` with a\n  bare-call fallback; a factor is confirmed only on a majority of successful\n  votes. Results are cached (model, prompt version, votes, task, evidence) so the\n  metric/reward double-read costs one consultation and rewards are repeatable.\n- **Fail-loud.** On a majority judge failure the `judge_error` metric is set to 1.0\n  and, under `judge_strict` (default), the rollout's factor credit is withheld\n  (never silently scored ungated); `judge_strict=False` keeps the deterministic\n  credit but still flags `judge_error`.\n- **Injection-fenced, versioned prompt** (`<fact_record>` fencing;\n  `JUDGE_GATE_PROMPT_VERSION`), and 0-weight telemetry metrics `factor_accuracy_lexical`\n  (pre-gate), `judge_gated`, `judge_error`.\n- **Sync/async boundary.** verifiers v1 scores the synchronous Taskset metrics from\n  inside a running event loop, so the async judge is driven to completion on a\n  dedicated worker thread with its own loop (`run_gate_blocking`); with no running\n  loop it uses `asyncio.run`. Both paths are covered by a regression test.\n\nValidation status: **pattern-ported from agent-liability-gym's validated gate,\nper-env live validation pending.** The confirm-all / reject-all / fabricated-quote\n/ majority-vote / fail-loud / r-mode paths are unit-covered with a stub judge, but\nthe gate has not yet been run against a live judge model on this env's held-out\nsplit. Default (no `judge_model`) behavior is byte-identical to v0.4.x.\n\n## Known Issues\n- **~85% keyword-solvable (default reward).** A per-factor 5-branch string classifier with zero\n  legal reasoning scores 0.8531 of max task_reward on the held-out split. `factor_accuracy`\n  (0.40 weight, the largest term) never checks that the cited quote/connection pertains to its\n  factor in the default deterministic path, and the anti-gaming apparatus in scoring.py\n  (`_has_incompatible_route_cue`, `_FACTOR_EXPLANATION_PATTERNS`) is dead code, never called. Do\n  not train unsupervised against the default reward. The v0.5.0 judge gate (above) is the\n  intended fix but is off by default and not yet live-validated on this env.\n- **verifiers pin.** This env imports `vf.TasksetConfig`, which exists in verifiers 0.1.14 but\n  NOT in PyPI 0.2.x — local testing requires `verifiers==0.1.14`; confirm the hosted image's\n  verifiers build before running hosted evals.\n- v0.4.0 (validate-don't-repair) changes scoring semantics: pre-0.4.0 eval numbers are not\n  comparable (the 0.3.0 canonicalizer repaired responses before scoring; the trained-vs-GPT-5.4\n  head-to-head inverts from +0.0011 to −0.0792 under raw scoring — see HOWEY_RETRACTION.md).\n\n## Semantic break at v0.4.0\n\nScores from <= 0.3.0 are not comparable to v0.4.0 and later. Through 0.3.0 the\nharness *repaired* each response before scoring: a canonicalizer appended a\nrewritten assistant message that overwrote the model's stated per-factor\ndisposition (from its connection choice), conclusion (from the factor labels),\nand missing_facts (from the \"unclear\" factors), and injected a fixed reasoning\nsentence and a constant disclaimer. Every metric then scored that repaired\npayload, so conclusion accuracy, factor accuracy, missing-fact F1, and the\ndisclaimer check did not measure the model's actual output. In addition,\nauthority precision punished any citation outside a narrow keyword-cue allowlist,\nso legitimate supplementary citations of real closed-universe authorities lost\ncredit unless the prompt happened to contain a cue phrase. v0.4.0 scores the raw\nresponse, adds a `route_consistency` metric, makes authority precision tolerant\nof any in-universe supplementary citation (with a structural cap of three\n`authority_ids`), and rebalances the reward. A model that relied on the old\nrepair or the authority allowlist will score differently.\n- Holdout policy: any observed blind set is consumed diagnostic evidence only. A new deploy-time blind set is independently authored only after the data, scorer, authority card, one-turn harness, selected checkpoint, deployment behavior, and competition thresholds are frozen. It is not shipped in this package.\n\n## Data provenance and limits\n\nAll scenarios are deterministic synthetic combinations of hand-authored fact\nclauses and coherence constraints. No client records, scraped legal questions,\nbenchmark answers, or model-generated labels are used. Train/dev/test reserve\ndifferent semantic graphs and wording families, but synthetic success does not\nestablish performance on live matters, new jurisdictions, adversarial facts, or\nlaw outside the supplied authority card.\n\n## Quickstart\n\n```bash\nprime env install crypto-law-howey\nprime eval run crypto-law-howey -m Qwen/Qwen3.5-2B -n 20 -r 1 -s -C task\nprime eval run crypto-law-howey -m gpt-5.5 -p openai --api-client-type openai_responses -n 20 -r 1 -s -C task\n```\n\nPublic diagnostic split:\n\n```bash\nprime eval run crypto-law-howey \\\n  -a '{\"config\":{\"taskset\":{\"split\":\"test\",\"eval_size\":80},\"harness\":{\"max_turns\":1}}}' \\\n  -n 80 -r 1 -s -C task\n```\n\nThe final parity verifier compares runs with identical environment source, task\nIDs, prompt hash, sampling, and harness configuration. It fails closed on missing\ntask payloads, run errors, protocol drift, or a trained-model schema rate below\n99%.\n\n## Changelog\n\n### 0.6.0 (2026-07-23) — deterministic grounding gate closes the keyword-classifier hole\nThe hardpass C5 exploit (a zero-reasoning per-factor keyword classifier scored ~0.85 of\ntask_reward with guessed labels + non-pertinent quotes) is CLOSED. Every substantive reward\nterm now requires the model's factor analysis to be grounded in the fact record:\n- `factor_accuracy` (0.40): a factor's label match credits only when its cited evidence grounds\n  it (>=0.75, the `complete_grounding` bar) — label-guessing without pertinent quotes pays 0.\n- `conclusion_accuracy` (0.22), `route_consistency` (0.07), `authority_f1` (0.05),\n  `missing_facts_f1` (0.13): scaled by `grounded_fraction` (fraction of factors grounded >=0.75),\n  so a self-consistent but ungrounded guess earns nothing from them. New 0-weight telemetry\n  metric `grounded_fraction` makes the scale observable; `factor_accuracy_lexical` remains the\n  pre-gate label-only recall. Raw-math helpers (`_authority_f1_raw`, `_missing_facts_f1_raw`)\n  are used by isolation unit tests; the public metrics are grounding-scaled.\n\n**Measured:** the refined keyword-sniff attack (no LLM, guessed labels + memorized authority\ntable + non-pertinent quotes) fell from **0.8531 -> 0.0300** (the residual is the flat\n`disclaimer_ok` string every response gets). A fully-grounded correct answer still scores 1.0;\na 3-of-4-grounded answer scores ~0.76. **Reward-semantics change:** pre-0.6.0 scores are NOT\ncomparable — partially-grounded and keyword-only answers score much lower now. 121 tests green.\n\n<!-- prior -->\n## Changelog\n\n- **0.5.0** (2026-07-22): optional off-by-default judge GATE on `factor_accuracy` (see Judge\n  gate). When `judge_model` is supplied, each lexically credited factor keeps its credit only if\n  an LLM judge confirms the cited `evidence_quote` pertains to that factor, verified by a\n  mechanical verbatim-substring check; majority vote, `temperature=0`, in-process cache,\n  fail-loud (`judge_error` + strict withholding), injection-fenced versioned prompt, and 0-weight\n  telemetry (`factor_accuracy_lexical`, `judge_gated`, `judge_error`). The sync Taskset metric\n  drives the async judge on a worker thread (`run_gate_blocking`). Default (no `judge_model`)\n  scoring is byte-identical to 0.4.x — the 104 existing tests pass unchanged; 16 stub-judge tests\n  added. Pattern-ported from agent-liability-gym's validated gate; per-env live validation pending.\n  Fix (pre-publish, same 0.5.0): client-construction failure in the gate — most commonly an\n  unset or misnamed `JUDGE_API_KEY` — now routes through the SAME fail-loud path as a judge-call\n  failure (`judge_error=1.0` + strict withholding + a stderr diagnostic that names the missing env\n  var) instead of escaping as an unhandled exception mid-scoring; a regression test covers the\n  unset-key path (strict withholds, non-strict keeps deterministic credit). r-mode stays\n  byte-identical; total test count 121.\n- **0.4.2** (2026-07-21): `load_environment` tolerates unknown kwargs (hosted CI probes with\n  `max_turns`, which a single-turn Taskset env rejects on signature alone).\n- **0.4.1** (2026-07-21): pin `verifiers>=0.1.14,<0.2.0`. Prime's hosted CI image resolves an\n  unpinned requirement to verifiers 0.2.x, where `vf.TasksetConfig` does not exist — the 0.4.0\n  push failed CI on import. No code changes.\n- **0.4.0**: validate-don't-repair scoring redesign (see Known Issues and HOWEY_RETRACTION).\n","encoding":"utf-8","truncated":false,"total_bytes":13092},"status":null}