{"data":{"kind":"file","path":"README.md","version_id":"jigbqcz0gm7lp7gk1hejawi9","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":11252,"modified_at":"2026-07-23T04:36:59.817000","content_hash":"167e6920c03d55b4e7abc0598f1f9256027da68cb96beb15edfab9e074ce7871"},"entries":[],"content":"# market-structure-schema-discipline\n\nM8 rails harness: Market Structure Impact Mapper output discipline — valid JSON, required fields, no categorical exemptions, grounded citations, proposed-bill caveat.\n\n### Task\nSingle-turn. The model emits one JSON object; the rubric scores it with\ndeterministic reward functions — no lawyer labels. An **optional** LLM judge\n*gate* (off by default) can be layered on top of the citation-grounding\ncredit; see Environment Arguments and Changelog 0.2.0.\n\n### Datasets\nSynthetic, templated market-structure scenarios embedded in the module\n(gzip+base64). Splits selected via the `split` environment argument.\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `split` | str | `\"train\"` | `train` or `eval` |\n| `max_examples` | int | `-1` | cap on dataset size (-1 = all) |\n| `judge_model` | str \\| None | `None` | If set, activates the judge gate over citation-grounding credit. **Omit for byte-identical default scoring.** |\n| `judge_base_url` | str \\| None | `None` | OpenAI-compatible base URL for the judge client |\n| `judge_api_key_var` | str | `\"JUDGE_API_KEY\"` | env var holding the judge API key |\n| `judge_gate` | bool | `True` | keep the judge as a remove-only gate (set `False` to disable gating even when a model is passed) |\n| `judge_strict` | bool | `True` | on judge failure, withhold the gated credit (`False` keeps the lexical value, still flags `judge_error`) |\n| `judge_votes` | int | `3` | majority-vote judge calls per rollout (temperature=0) |\n| `judge_max_retries` | int | `2` | per-vote retries on parse/API failure |\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | mean of the rubric's reward functions (each in [0,1]) |\n| `citations_pre_gate` | (judge-gated runs only, weight 0) pre-gate `reward_citations_grounded` value — the lexical/deterministic score before the judge removes anything |\n| `citations_post_gate` | (judge-gated runs only, weight 0) post-gate value — identical to the gated `reward_citations_grounded` |\n| `judge_gated` | (judge-gated runs only, weight 0) 1.0 when the gate had at least one lexically-grounded citation to evaluate for this rollout |\n| `judge_error` | (judge-gated runs only, weight 0) 1.0 when a majority of judge votes failed after retries — an infra signal, not a capability signal |\n\nPart of the M8 Market Structure Impact Mapper \"rails\" RFT harness.\n\n### Changelog\n- **0.2.1** (2026-07-22): Fixed a live-validated judge-gate wiring defect that\n  taxed genuinely grounded, matter-specific citations to 0 whenever the\n  completion's `§` / `¶` arrived JSON-escaped. `_gate_citations_rollout`\n  built the citation-extraction candidates from `_unescape_symbols(text)`\n  (correctly handling `json.dumps()`'s default `ensure_ascii=True`, which\n  renders `§`/`¶` as literal `§`/`¶` ASCII sequences) but then\n  passed the RAW, still-escaped `text` into `judge_gate_confirm` — i.e. into\n  both the `<completion>` block the judge reads and the deterministic\n  verbatim-quote substring check. A judge naturally reproduces the real `§`\n  symbol when quoting a faithful, matter-specific span back (that is what a\n  correct copy of an escaped citation looks like once an LLM reads and\n  re-quotes it), so that quote then never substring-matched the still-escaped\n  raw completion, and `verify_judge_gate_payload` dropped the confirmation as\n  a \"fabricated quote\" — every time, for every genuinely grounded citation in\n  an escaped completion. Live-validated against `grok-4.5` (`judge_votes=3`)\n  before the fix: honest, matter-specific completions scored\n  `citations_pre_gate=1.0 -> citations_post_gate=0.0` on 5/5 probed rows (a\n  full 100% false-rejection, not a partial tax). `_gate_citations_rollout`\n  now unescapes once (`judge_text = _unescape_symbols(text)`) and uses that\n  same normalized text for extraction, the cache key, and the judge call.\n  Re-validated live post-fix: honest completions now score\n  `pre=1.0 -> post=1.0` (tax = 0.0000) on the same rows, and the gate still\n  correctly zeroes a genuine junk/name-dropped citation\n  (`pre=1.0 -> post=0.0`) — remove-only behavior is intact, only the false\n  rejection is gone. Regression-tested, network-free, with a `FakeJudge`\n  that returns the real (unescaped) symbol in its `completion_quote` exactly\n  as a live judge does\n  (`test_gate_confirms_matter_specific_citation_in_json_escaped_completion`,\n  plus a companion negative case\n  `test_gate_still_rejects_fabricated_quote_on_escaped_completion` proving\n  the fix did not turn the gate into a rubber stamp). Does not change any\n  existing test's expected value — every pre-0.2.1 fixture used the literal\n  `§`/`¶` character directly (never JSON-escaped), so none of them exercised\n  this path.\n- **0.2.0** (2026-07-22): Judge gate (pattern-ported from\n  agent-liability-gym; validation status: **live-validated against\n  `grok-4.5`, 2026-07-22 — see 0.2.1** for a defect the live run surfaced and\n  fixed). Optional, off by default. When `judge_model` is\n  passed, an LLM judge is applied INSIDE `reward_citations_grounded` as an\n  AND-gate over the per-citation credit the lexical section-existence\n  matcher already granted: for every distinct cited section id the lexical\n  matcher counted as \"grounded\" (a real section listed in\n  `info[\"source_sections\"]`), the judge confirms the completion's use of\n  that citation actually pertains to the request (matter-specific analysis,\n  not a bare name-dropped section number). It can only **REMOVE** credit —\n  post-gate grounded-fraction ⊆ pre-gate by construction, since candidates\n  are built exclusively from citations that already passed the lexical\n  existence check. Each confirmation is verified deterministically: the\n  judge must return a verbatim contiguous span copied from the MODEL'S\n  COMPLETION, checked as a normalized substring, so a hallucinated quote\n  loses that slot. Majority vote (`judge_votes`, default 3, temperature=0\n  with a bare-call fallback), in-process cache keyed\n  `(model, prompt-version, votes, candidate citations, question, text)`,\n  injection-fenced versioned prompt (`JUDGE_GATE_PROMPT_VERSION`). Fail-loud:\n  on majority judge failure `judge_error=1.0` and — under `judge_strict`\n  (default) — the gated credit is withheld (never silently scored ungated).\n  The reward path is already async, so the gate is awaited directly (no\n  event-loop helper needed). Telemetry metrics (`judge_gated`, `judge_error`,\n  `citations_pre_gate`, `citations_post_gate`) are exposed at weight 0.\n  **With no `judge_model` the environment is byte-identical to 0.1.4**\n  (existing regression tests pass unchanged). Stub tests\n  (`tests/test_judge_gate.py`) cover credit removal, no-add, fabricated-quote\n  rejection, 2-of-3 majority, fail-loud, and pre-gate == deterministic-scorer\n  — all network-free with a fake judge client.\n\n  Also fixed, unconditionally (not gated behind `judge_model`): `CITE_RE`'s\n  char class greedily included a bare TRAILING sentence period in the\n  captured citation identifier (e.g. `\"...under §202.\"` captured `\"202.\"`\n  instead of `\"202\"`), so the normalized capture no longer substring-matched\n  `info[\"source_sections\"]` and a genuinely, correctly cited real section\n  was silently scored as ungrounded — dropping credit for a legitimate\n  citation purely because of where a sentence ends. The capture group now\n  only consumes a `.` when it is followed by another identifier character\n  (i.e. mid-identifier, as in `\"240.17a-4\"`), never a bare trailing period.\n  Regression-tested\n  (`test_reward_citations_grounded_trailing_period_not_swallowed`,\n  `test_cite_re_still_captures_internal_period_in_compound_id`); does not\n  change any existing test's expected value (no existing fixture triggers\n  the trailing-period case).\n- **0.1.3** (2026-07-16): Fixed 3 confirmed defects from source-verified audit.\n  1. `last_text()` only recognized message dicts (`{\"content\": ...}`); a\n     completion whose last message was an object exposing `.content` as an\n     attribute (e.g. an OpenAI/pydantic `ChatCompletionMessage`) fell through\n     to `str(completion)`, repr-mangling valid JSON and scoring\n     `reward_valid_json` / `reward_required_fields` as 0.0 even on correct\n     output. Now duck-types `.get(\"content\")` or `getattr(..., \"content\")`\n     before giving up.\n  2. `reward_citations_grounded` returned 1.0 for zero citations — identical\n     to the score for citing every source correctly. Omission must not\n     strictly dominate grounded citation; zero citations now score 0.5.\n  3. Rubric weights `[1, 1, 1, 1, 1, 0.5]` summed to 5.5, contradicting the\n     \"mean of the rubric's reward functions (each in [0,1])\" claim above.\n     Weights are now divided by their sum so the rubric total is bounded in\n     [0, 1], with the original relative weighting (format = half of every\n     other component) unchanged.\n- **0.1.4** (2026-07-18): Audit fix. (1) `last_text()` now falls back to\n  `reasoning_content` when `content` is empty/None, for both dict and\n  object message shapes — closes a silent 0.0 for genuinely correct\n  reasoning-model completions. (2) `reward_required_fields` now requires\n  each required key to carry a non-empty value (not `\"\"`/`null`/`[]`/`{}`),\n  not just be present — an all-stub object dropped from 0.8139 to 0.6316.\n\n### Known Issues (disclosed, not fixed)\n- **Identity/grounding gap**: none of the six reward functions verify the\n  completion is actually about the question's stated company/product/\n  statute. A regex template that copies `PROVIDED SECTIONS:` and `status:`\n  straight out of the model-visible prompt scores 1.0000 — bit-identical to\n  the honest baseline. A fixed generic boilerplate template (same string\n  every row) scores a flat 0.9091. Do not treat a high score as evidence of\n  legal reasoning; do not train unsupervised against this reward without a\n  stronger identity/grounding check. The 0.2.0 judge gate (`judge_model` set)\n  narrows this gap for the **citation-grounding component only**: a judge\n  should reject a bare `PROVIDED SECTIONS:` copy-paste as not \"substantively\n  tied to the request\" and remove that credit. It does **not** touch the\n  other five reward functions (`reward_valid_json`,\n  `reward_required_fields`, `reward_no_forbidden_phrase`,\n  `reward_caveat_present`, `reward_format`), which stay fully deterministic\n  and remain just as exploitable by the template-copy attack described\n  above. The gate's real-world effect on this exploit **was live-validated\n  against `grok-4.5`** (2026-07-22, see Changelog 0.2.1): a `PROVIDED\n  SECTIONS:`-copy junk completion scores `citations_pre_gate=1.0 ->\n  citations_post_gate=0.0` live (fully removed), while a genuinely\n  matter-specific honest completion citing the same sections stays at\n  `1.0 -> 1.0` (zero tax) — after fixing a JSON-escape wiring defect the\n  live run surfaced (0.2.1). Even so, the gate narrows the identity/grounding\n  gap for the citation-grounding component only; the other five reward\n  functions listed above remain untouched and unvalidated against this\n  exploit. Do not read a judge-gated score as a fully solved\n  identity/grounding gap.\n","encoding":"utf-8","truncated":false,"total_bytes":11252},"status":null}