{"data":{"kind":"file","path":"README.md","version_id":"ut0qx8o76picmt6nae06wfy8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3531,"modified_at":"2026-07-19T20:45:07.209000","content_hash":"3b5b32b63ba25b5da59b00f6a7f700f791ed4011ca9121253084d7ed38f23652"},"entries":[],"content":"# coverageproof-env\n\n**Certified document-QA as an RL environment — the audit trail is the reward function.**\n\nA [verifiers](https://github.com/PrimeIntellect-ai/verifiers)-compatible\n(`0.2.0`, tested) single-turn environment over certified, machine-recheckable\ndocument-QA rows. Every row's reward is computable by anyone, from the row\nitself — no trust in the publisher required.\n\n## What it trains (and why it's hard to get elsewhere)\n\n**Two-sided grounding calibration.** Most grounding/hallucination data is\none-sided: answer accuracy OR abstention. This env mixes both families in one\ndataset, so total reward is only maximized by a model that is *calibrated*:\n\n- **Extractive rows** (needle, from fresh SEC filings): reward = exact\n  normalized match (1.0) with a token-F1 partial (0.5 weight) against\n  hash-pinned gold spans. Abstaining scores ~0 — over-abstention is punished.\n- **Absence rows** (verified-absent facts): reward = 1.0 only for a correct\n  abstention. Any invented answer scores 0 — hallucination is punished.\n\nA model that always answers gets ~half the total; a model that always\nabstains gets ~half. Only \"answer when supported, abstain when not\" wins.\nThat is the exact behavior RAG, grounding, and abstention-RL programs are\ncurrently buying data for.\n\n## Baselines (measured 2026-07-19, this env, verifiers 0.2.0)\n\nLocal 30B-class MoE engine (temperature 0, thinking on), 12 shuffled rows:\n\n| family | n | mean reward |\n|---|---:|---:|\n| absence | 4 | **1.000** |\n| extractive | 8 | **0.779** (0.33–1.00) |\n\nThe spread is the point: the reward discriminates tight span-level grounded\nanswers (1.0) from near-miss paraphrase (0.3–0.5) and verbose or wrong ones\n(~0.01 measured on reasoning-preamble completions before the parser view).\n\n## Data\n\nv0 ships 600 candidate rows (300 needle + 300 absence) built from SEC EDGAR\nfilings dated **2026-07-01 → 2026-07-15** (post-cutoff fresh split —\ncontamination-impossible by construction), 150/171 distinct source documents,\neach row carrying `doc_ref` to the exact public filing. Rows are\nquarantine-grade candidates from SovNodeAI's certified-document-QA factory\n(span-verified answers; absence rows screened by deterministic\nformat-variant sweeps + model abstention probes). A CoverageProof insurance\nlane (600 more rows incl. multihop) is staged for v0.2.\n\n## Usage\n\n```python\nimport coverageproof_env\n\nenv = coverageproof_env.load_environment()          # full 600-row set\n# env = coverageproof_env.load_environment(limit=100)\n```\n\nEvaluate any OpenAI-compatible endpoint:\n\n```python\nfrom verifiers.types import ClientConfig\n\ncfg = ClientConfig(client_type=\"openai_chat_completions\",\n                   api_base_url=\"http://localhost:8002/v1\",\n                   api_key_var=\"LOCAL_KEY\")\nout = env.evaluate_sync(cfg, model=\"your-model\", num_examples=100,\n                        rollouts_per_example=1, shuffle=True)\n```\n\n## Reward details\n\n- Completions are read from the final assistant message(s); scoring takes the\n  better of the whole completion vs its final paragraph, so both concise and\n  think-then-answer styles are scored on the answer, not the reasoning.\n- Absence detection: calibrated abstention-pattern match on the completion.\n\n## Provenance\n\nRows: `generator: kimi-k3-swarm` candidates from the SovNodeAI certified\ndocument-QA program (public audits: preregistrations, hash-bound ledgers,\ncorrection chains). License/provenance review pending the K3 open-weights\nrelease — see the dataset card before redistribution.\n","encoding":"utf-8","truncated":false,"total_bytes":3531},"status":null}