{"data":{"kind":"file","path":"README.md","version_id":"tv2k5aze646fygqnj4d7qiop","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7658,"modified_at":"2026-07-23T04:03:05.073000","content_hash":"56bc847b175a01dbded4dda66aa7c2fb5568888d27d99ea285c282d0a5ba124a"},"entries":[],"content":"# market-structure-missing-facts\n\nM8 rails harness: missing-fact detection — flag withheld facts, decline to conclude on incomplete facts, and do not over-ask when every needed fact is present.\n\n### Task\nSingle-turn. The model emits one JSON object; the rubric scores it with\ndeterministic reward functions — no lawyer labels. An **optional** LLM judge\n*gate* (off by default) can be layered on top of the per-fact recall credit; see\nEnvironment Arguments and Changelog 0.2.0.\n\n### Datasets\nSynthetic, templated market-structure scenarios embedded in the module\n(gzip+base64). Splits selected via the `split` environment argument.\n\n### Environment Arguments\n| Arg | Type | Default | Description |\n| --- | ---- | ------- | ----------- |\n| `split` | str | `\"train\"` | `train` or `eval` |\n| `max_examples` | int | `-1` | cap on dataset size (-1 = all) |\n| `judge_model` | str \\| None | `None` | If set, activates the judge gate over per-fact recall credit. **Omit for byte-identical default scoring.** |\n| `judge_base_url` | str \\| None | `None` | OpenAI-compatible base URL for the judge client |\n| `judge_api_key_var` | str | `\"JUDGE_API_KEY\"` | env var holding the judge API key |\n| `judge_gate` | bool | `True` | keep the judge as a remove-only gate (set `False` to disable gating even when a model is passed) |\n| `judge_strict` | bool | `True` | on judge failure, withhold the gated credit (`False` keeps the lexical value, still flags `judge_error`) |\n| `judge_votes` | int | `3` | majority-vote judge calls per rollout (temperature=0) |\n| `judge_max_retries` | int | `2` | per-vote retries on parse/API failure |\n\n### Metrics\n| Metric | Meaning |\n| ------ | ------- |\n| `reward` | weighted sum of the rubric's reward functions, normalized to [0,1] |\n| `reward_missing_facts_recall` | per-fact recall of withheld facts (judge-gated when `judge_model` set) |\n| `recall_pre_gate` | (judge mode, weight 0) lexical recall before the gate |\n| `recall_post_gate` | (judge mode, weight 0) recall after the gate — always ≤ `recall_pre_gate` |\n| `judge_gated` | (judge mode, weight 0) 1.0 when the gate is active for the rollout |\n| `judge_error` | (judge mode, weight 0) 1.0 when the judge could not be consulted (infra failure, not model failure) |\n\nPart of the M8 Market Structure Impact Mapper \"rails\" RFT harness.\n\n### Changelog\n- **0.2.1** — Doc-only: Known Issues corrected to record that a live\n  gaming-pass (2026-07-22) found the judge gate does not mitigate the\n  frequency-bias spray exploit.\n- **0.2.0** — Judge GATE (opt-in, off by default; pattern ported from\n  agent-liability-gym's engine per the 2026-07-22 judge-port directive). When a\n  `judge_model` is passed to `load_environment`, an LLM judge is applied INSIDE\n  `reward_missing_facts_recall` as an AND-gate over the per-fact recall credit:\n  for every `(withheld_fact ↔ model_stated_fact)` pair the fuzzy keyword matcher\n  counted as covered, the judge confirms the model's stated missing fact IS the\n  same withheld fact (not a keyword coincidence). It can only **REMOVE** credit —\n  post-gate recall ⊆ pre-gate recall by construction — closing the\n  Known-Issue frequency-bias spray (a constant two-fact list keyword-matches but\n  is not a scenario-specific treatment). Each confirmation is verified\n  deterministically: the judge must return a verbatim contiguous span copied from\n  the MODEL OUTPUT, checked as a normalized substring, so a hallucinated quote\n  loses that slot. Majority vote (`judge_votes`, default 3, temperature=0 with a\n  bare-call fallback), in-process cache keyed\n  `(model, prompt-version, votes, candidate pairs, text)`, injection-fenced\n  versioned prompt (`JUDGE_GATE_PROMPT_VERSION`). Fail-loud: on majority judge\n  failure `judge_error=1.0` and — under `judge_strict` (default) — the gated\n  credit is withheld (never silently scored ungated). The reward path is already\n  async, so the gate is awaited directly (no event-loop helper needed). Telemetry\n  metrics (`judge_gated`, `judge_error`, `recall_pre_gate`, `recall_post_gate`)\n  are exposed at weight 0. **With no `judge_model` the environment is\n  byte-identical to 0.1.4** (existing regression tests pass unchanged).\n  Validation status: pattern-ported; per-env live validation pending.\n  Stub tests (`tests/test_judge_gate.py`) cover credit removal, no-add,\n  fabricated-quote rejection, 2-of-3 majority, fail-loud, and pre-gate ==\n  deterministic-scorer — all network-free with a fake judge client.\n- **0.1.4** — Audit fix (2026-07-17). (1) `last_text` now also checks a\n  `reasoning_content` field (dict key or attribute) when `content` is\n  empty/None, matching the pattern some reasoning-model SDK paths use\n  (e.g. Qwen3.5 puts the real answer in `reasoning_content`); previously a\n  genuinely correct completion in that shape stringified to `\"\"`, scored\n  0.0 reward, and was non-deterministic across SDK/pretty-printing paths.\n  (2) `reward_missing_facts_recall` and `reward_no_premature_conclusion` no\n  longer return their `all_present=True` credit (1.0) before parsing the\n  completion; they now require `extract_json(last_text(completion))` to\n  succeed first. This closes a floor exploit where an empty string or\n  garbage completion banked 0.5714/1.0 (57%) of max reward on `all_present`\n  rows (49% of the dataset) for free. A genuinely correct completion is\n  unaffected (still 1.0).\n- **0.1.3** — Audit fix. (1) `last_text` now duck-types the last completion\n  message: it also accepts message-like objects exposing a `.content`\n  attribute, not only plain dicts. (2) `reward_no_premature_conclusion` and\n  `reward_no_overask` no longer pay out on hedging alone (always refusing to\n  conclude / trivially-true when facts are withheld) — both now require the\n  model's `missing_facts` to fuzzy-match at least one gold withheld fact\n  (the same matching rule as `reward_missing_facts_recall`), so a\n  content-free hedging shell no longer banks ~71% of max reward. (3) Added\n  `firm`, `facts`, `material`, `regulatory` to `STOPWORDS` (generic terms\n  that were letting vague answers coincidentally overlap short gold facts),\n  and fact matching now requires whichever is stricter of >=2 matched\n  content keywords or >=50% keyword overlap, closing the single-word-\n  coincidence loophole on short (2-3 keyword) gold facts. (4) Rubric\n  weights normalized from `[1, 1, 1, 0.5]` (summed to 3.5) to\n  `[2/7, 2/7, 2/7, 1/7]` (sums to 1.0, same relative proportions) so total\n  reward is in [0,1].\n\n### Known Issues (disclosed, not fixed)\n- **Frequency-bias spray**: the fuzzy keyword-overlap fact matcher can be\n  satisfied by a fixed, scenario-independent list of the two most common\n  withheld facts in the training distribution (\"the customer type\", \"the\n  jurisdictions of operation\"). Applied identically to every row, this\n  constant list scores 0.7274 (73% of the 1.0 honest baseline) without\n  reading any scenario text. Do not train unsupervised against this\n  reward without a less frequency-skewed / more scenario-specific matcher.\n- **Judge gate does not close the frequency-bias spray**: Live gaming-pass\n  2026-07-22 (grok-4.5, 20 rows): the judge gate does NOT reduce the constant\n  frequency-spray exploit (0.725 pre == 0.725 post, zero honest tax). The\n  spray's hits are rows whose gold withheld fact IS the majority-class\n  answer, so \"same fact\" confirmation is genuinely true — outcome-only\n  judging cannot detect scenario-blind guessing. Mitigation requires a\n  schema-level change (per-fact scenario-grounding quotes) or dataset\n  rebalancing; both are open design work. Do not treat the gate as\n  protection against this exploit.\n","encoding":"utf-8","truncated":false,"total_bytes":7658},"status":null}