{"data":{"kind":"file","path":"README.md","version_id":"taw38him2sr8jtnzv0y9vkzq","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7050,"modified_at":"2026-07-23T05:53:14.273000","content_hash":"0ec5a89c0bb0b97fc7464a718ec32d0f1016af2618f9b520395161f665945dcd"},"entries":[],"content":"# market-structure-source-status\n\nM8 rails harness: legal-source status classification — bill vs enacted law vs proposed/final rule vs guidance vs enforcement action; never treat proposed text as binding law.\n\n### Task\nSingle-turn. The model emits one JSON object; the rubric scores it with\ndeterministic reward functions — no LLM judge, no lawyer labels.\n\n### Datasets\nSynthetic, templated market-structure scenarios embedded in the module\n(gzip+base64). Splits selected via the `split` environment argument.\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `split` | str | `\"train\"` | `train` or `eval` |\n| `max_examples` | int | `-1` | cap on dataset size (-1 = all) |\n| `judge_model` | str \\| None | `None` | **OFF BY DEFAULT.** Set to enable the judge gate over source_type/is_binding_law credit. With this unset, behavior is byte-identical to 0.1.4. |\n| `judge_base_url` | str \\| None | `None` | Optional OpenAI-compatible base URL for the judge client. |\n| `judge_api_key_var` | str | `\"JUDGE_API_KEY\"` | Env var holding the judge API key (required, fail-loud, when the gate is active). |\n| `judge_gate` | bool | `True` | Master gate switch; `False` restores plain lexical scoring even if `judge_model` is set. |\n| `judge_strict` | bool | `True` | On judge failure, withhold all status credit for the rollout (visible collapse) instead of silently falling back to lexical credit. |\n| `judge_votes` | int | `3` | Parallel judge calls per gated rollout; a field confirms only on majority. |\n| `judge_max_retries` | int | `2` | Retries per vote before that vote counts as failed. |\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | weighted sum of the rubric's reward functions; weights sum to 1.0, so `avg_score` reads as a fraction in [0, 1] |\n| `reward_source_type_lexical` / `reward_binding_correct_lexical` | (judge-gate mode only, 0-weight) pre-gate lexical credit, for comparing against the post-gate `reward_source_type` / `reward_binding_correct` |\n| `judge_gated` | (judge-gate mode only, 0-weight) 1.0 when the gate was applied to this rollout |\n| `judge_error` | (judge-gate mode only, 0-weight) 1.0 when the gate judge could not be consulted after retries — under `judge_strict` this rollout's status credit was withheld; treat a spike as infra failure, not a capability signal |\n\nPart of the M8 Market Structure Impact Mapper \"rails\" RFT harness.\n\n### Judge Gate (v0.2.0, opt-in)\nPattern-ported from agent-liability-gym's validated judge-gate protocol\n(`JUDGE_GATE_PROMPT_VERSION` / `_JUDGE_GATE_CACHE` / `build_judge_gate_prompt`\n/ `verify_judge_gate_payload` / `judge_gate_confirm`, majority vote,\ntemperature=0, in-process cache). When `judge_model` is passed, the judge\nbecomes an AND-gate over the \"status credit\" (`reward_source_type` +\n`reward_binding_correct`): it can only REMOVE credit the deterministic\nlexical match already granted, never add it. For each lexically-credited\nfield, the judge is shown `info[\"source_text\"]` (the actual legal-source\ndescription, previously never read by any reward function in this env —\nsee \"Known Issues\" below) and must return a verbatim quote from that text\njustifying the field; a quote that isn't a real substring of the source\ntext is mechanically discarded, so a hallucinating judge loses that slot.\nJudge failure is fail-loud: `judge_error=1.0` plus (under the default\n`judge_strict=True`) the rollout's status credit is withheld rather than\nsilently scored ungated.\n\n**Validation status: pattern-ported, per-env live validation pending** — this\nhas not yet been run against a live `judge_model`; the underlying protocol is\nvalidated in agent-liability-gym (0 FP / 0 FN, gate-v4 + votes=3, grok-4.5),\nbut this env's specific prompt/wiring has not had its own live pass yet.\n\n### Changelog\n- 0.2.0: Added an opt-in judge gate (off by default; `judge_model=None`\n  reproduces 0.1.4 byte-for-byte) over the source_type/is_binding_law status\n  credit. The judge confirms each lexically-credited field against\n  `info[\"source_text\"]` — closing the disclosed identity gap for gated\n  rollouts, since no reward function otherwise reads that field — and can\n  only remove credit, never add it. Majority vote (default 3), in-process\n  cache, fail-loud on judge failure (`judge_strict` default withholds\n  credit). See \"Judge Gate\" section above.\n- 0.1.4: `last_text` now falls back to `reasoning_content` when `content`\n  is missing/empty, instead of returning the literal string `\"None\"` and\n  scoring a correct reasoning-model completion as 0.0. The 0.1.3 duck-type\n  fix stopped at \"is `.get` callable?\", but `verifiers.types.AssistantMessage`\n  (a Pydantic model) implements `.get(key, default)` as\n  `getattr(self, key, default)` — so `get(\"content\", \"\")` returned the\n  real (`None`) attribute value, not the `\"\"` default, and the function\n  never reached the `reasoning_content` fallback. Also fixes the same\n  `None` → `\"None\"` cosmetic bug for plain dict messages.\n- 0.1.3: `last_text` now duck-types the last completion message (accepts\n  any object exposing `.get`/`.content`, not only `dict`) instead of\n  silently scoring 0.0 on non-dict message objects. Rubric weights\n  renormalized from `[1.0, 1.0, 1.0, 0.5]` (sum 3.5) to\n  `[2/7, 2/7, 2/7, 1/7]` (sum 1.0) so `avg_score` is a fraction in [0, 1]\n  instead of up to 3.5; relative weighting between rubrics is unchanged.\n\n### Known Issues (disclosed, not fixed in default r-mode)\n- **Identity/grounding gap (default r-mode, `judge_model=None`)**:\n  `info[\"source_text\"]` (the actual legal-source description) is never read\n  by any of the four deterministic reward functions. Two different bills,\n  scored with one description swapped for the other, produce an identical\n  reward (1.0000 = 1.0000). A constant non-binding label applied to every\n  row (never reading the source text) reaches 0.6667 (67% of honest).\n  `reward_no_premature_obligation` pays 1.0 to any text lacking five fixed\n  keyword patterns, including the empty string (28.6% of the rubric banked\n  for free on every row). Do not train unsupervised against this reward\n  without a grounding check tying the completion to info[\"source_text\"] —\n  **the opt-in judge gate above is that check** for the status-credit portion\n  (source_type/is_binding_law) when `judge_model` is set; the\n  `reward_no_premature_obligation` free-credit hole and `reward_format`\n  shaping reward are untouched by the gate either way.\n\n**Known gold-label error (live-verified 2026-07-22).** One eval row's `source_text` states the\nmeasure \"has not been formally introduced as a bill\" while its gold `source_type` is `bill` —\nthe judge gate correctly rejects the gold label when reading the source. Until the embedded\ndataset blob is regenerated, treat a single source_type miss on an otherwise-correct honest\nrun as this row, not a model error. (Dataset is a baked base64+gzip blob; in-place surgery\ndeferred to the next dataset regeneration rather than risking silent corruption.)\n","encoding":"utf-8","truncated":false,"total_bytes":7050},"status":null}