{"data":{"kind":"file","path":"README.md","version_id":"sos42d21n3sr7p3zyph66wiz","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":7959,"modified_at":"2026-08-30T09:39:41.240000","content_hash":"f2d89d8cd93dc5733ca5c3ef20579843fc1eff77289d4f30acc3f0c7a79e177f"},"entries":[],"content":"# commonground-elicit\n\n`commonground-elicit` 0.5.0 is an unreleased deterministic Verifiers candidate\nfor two structured tasks over fictional stakeholder-policy scenarios:\n\n- `find`: localize and diagnose planted ambiguities, contradictions, and gaps.\n- `elicit-ask`: select the two highest-value clarification targets from three\n  planted issues and predict faction stances on each yes/no question.\n\nThe current public 0.4.1 artifact remains available for reproducibility but is\nsuperseded for benchmark use. Five `regional-archives-access` rows selected the\nwrong second contradiction passage, and public faction summaries used a finite\nphrase table keyed by issue type and exact target stance. Version 0.5 replaces\nboth mechanisms and requires a fresh Elicit study before promotion.\n\nThis environment is associated with [Context Engine](https://contextengine.sh),\nwhose organizational workflow can surface unresolved decisions and collect\nstakeholder responses. A future governed exporter could allow consenting\nindividuals and groups to retain, license, or sell derived preference data.\nVersion 0.5 is entirely synthetic and includes no such exporter.\n\n## Data and separation\n\n| Split | Rows | Templates × variants | Template/layout profile |\n| --- | ---: | --- | --- |\n| `train` | 100 | 4 × 25 | `train-template-layout-profile-v4` |\n| `eval` | 100 | 20 × 5 | `heldout-template-layout-profile-v4` |\n\nEvery row contains one ambiguity, contradiction, and gap plus three to five\nfactions. Each faction has a general value vector spanning access,\nadaptability, continuity, oversight, and safety. Its public summary renders\nthose values once without consulting any planted issue or target stance. Each\nissue separately defines a value trade-off and whether “yes” selects the\nprimary rule or an alternative. The hidden stance vector is recomputed from\nthat composition, so reversing question polarity also reverses agree/disagree.\n\nEvery contradiction template authors a stable opposing document and exact\nquote. Generation remaps opaque IDs but never infers the relationship from\nlexical overlap. Validation rejects a related passage that is absent, in the\nsame document, another planted anchor, or a distractor. Corpus-wide tests cover\nall 24 template families, including the corrected regional-archives phone-\nphotograph rule.\n\nGeneration separately audits exact instances, canonical visible propositions,\nand policy-issue semantics after removing opaque identity/layout fields. It\nblocks cross-split overlap for all relevant hashes and enforces bounded token-\nJaccard and word unigram/bigram TF-IDF neighbors. These deterministic checks\nare not an embedding proof of conceptual independence. Both profiles still\nshare one core synthetic generator.\n\n## Find contract\n\nReturn one diagnosis per suspected issue:\n\n```json\n{\n  \"findings\": [{\n    \"doc_id\": \"doc-a1b2c3d4\",\n    \"quote\": \"The copied primary passage.\",\n    \"type\": \"contradiction\",\n    \"diagnosis\": \"Should the practical exception override the primary rule?\",\n    \"related_evidence\": {\n      \"doc_id\": \"doc-e5f6a7b8\",\n      \"quote\": \"The copied conflicting rule.\"\n    }\n  }]\n}\n```\n\n`diagnosis` must be a well-formed yes/no question. It is not compared with a\nhidden canonical vocabulary: exact public evidence, issue type, and\nrelationship carry deterministic semantic grounding. Contradictions require a\ncontiguous second passage from another document; ambiguity and gap findings\nrequire `related_evidence: null`. Duplicate normalized spans invalidate the\nresponse, so repeating one anchor under all types cannot hedge.\n\nEnd-to-end matching requires the correct document and type, at least 90%\ncontiguous anchor coverage, at least 80% evidence-token precision, valid\ndiagnosis form, and the correct relationship. The environment reports strict\n`finding_f1` plus localization, type, diagnosis, and relationship diagnostics.\n`question_utility` remains a weight-zero companion signal.\n\n## Ask contract\n\nThe default row has three candidate issues and requires exactly two questions.\nEach item must identify the issue structurally and declare what “yes” means:\n\n```json\n{\n  \"questions\": [{\n    \"doc_id\": \"doc-a1b2c3d4\",\n    \"quote\": \"The copied issue passage.\",\n    \"type\": \"contradiction\",\n    \"question\": \"Should the practical exception override the primary rule?\",\n    \"yes_choice\": \"alternative\",\n    \"related_evidence\": {\n      \"doc_id\": \"doc-e5f6a7b8\",\n      \"quote\": \"The copied conflicting rule.\"\n    },\n    \"target_stances\": {\n      \"group-1a2b3c\": \"agree\",\n      \"group-4d5e6f\": \"disagree\"\n    }\n  }]\n}\n```\n\nQuestion prose must have yes/no form but is not compared with hidden authored\ntokens or aliases. Exact visible grounding, type, `yes_choice`, relationship,\nand complete faction stances determine semantic credit. If a model reverses\nthe question orientation and inverts agree/disagree consistently, it receives\nthe same stance credit. Separate metrics report question format validity,\ngrounding recall, and stance accuracy.\n\nUtility combines structured issue grounding, stance accuracy, panel\ndisagreement, and the share of factions whose composed position is non-pass.\nThere is no policy-keyword value table. Global assignment prevents duplicate\nclaims, and normalization by the best attainable two-item sum makes the exact\ntop-two response score 1.0. This is clarification-target selection—not a\nmeasurement of real-world information gain. Three candidates versus K=2\nremains a deliberately small ranking problem.\n\n## Model-free comparators\n\nExact results on the bundled 100-row 0.5 candidate evaluation split:\n\n| Comparator class | Task | Comparator | mean reward |\n| --- | --- | --- | ---: |\n| Prompt-observable | find | Random visible spans | 0.080 |\n| Prompt-observable | find | Flag vague-sounding spans | 0.195 |\n| Prompt-observable | find | Legacy 0.2 document-ID/position codebook | 0.000 |\n| Prompt-observable | elicit-ask | Template clarity questions | 0.078 |\n| Prompt-observable | elicit-ask | Randomly targeted questions | 0.069 |\n| Prompt-observable | elicit-ask | Removed 0.3 summary/stance codebook | 0.000 |\n| Component oracle | elicit-ask | Exact issues + removed 0.4 principle-table parser | 0.000 |\n| Source-aware prompt-only | elicit-ask | Public template detector + removed 0.4 principle-table parser | 0.000 |\n| Component oracle | elicit-ask | Exact top-K issues + random stances | 0.670 |\n| Component oracle | elicit-ask | Exact top-K issues + exact stances (ceiling) | 1.000 |\n\nThe historical 0.4 decoder is tested against a frozen old-format fixture, where\nboth the exact-issue component and public-template prompt detector reach 1.0,\nand against the 0.5 corpus, where both reach 0.0. Component oracles read hidden\nissue selection and are not prompt-observable floors.\n\n## Usage\n\n```bash\nuv run validate commonground-elicit --runtime.type subprocess --rich false\nuv run validate commonground-elicit --taskset.task-mode elicit-ask \\\n  --runtime.type subprocess --rich false\nuv run eval commonground-elicit -m MODEL --no-push\nuv run eval commonground-elicit --env.taskset.task-mode elicit-ask \\\n  -m MODEL --no-push\n```\n\nDifficulty controls are `docs_count`, `docs_length`, `planted_density`,\n`distractor_density`, `panel_polarization`, and `question_count`. Parsing is\nbounded and malformed output fails closed. Public planted keys support open\ntraining but not contamination-resistant comparison; use a fresh private\ngenerator family for consequential evaluation.\n\nFind additionally accepts `reward_mode=\"shaped\"` for training. It averages\nlocalization, type, diagnosis, and relation F1 scores on `[0,1]`, charging every\nunmatched candidate at every stage. Adding a false positive or overlapping\nhedge strictly lowers reward relative to a concise exact answer. The default is\n`reward_mode=\"strict\"`; reported evaluations must use it. Neither objective has\ndemonstrated learning value until multi-seed training and fresh-family transfer\nare run.\n","encoding":"utf-8","truncated":false,"total_bytes":7959},"status":null}