{"data":{"kind":"file","path":"README.md","version_id":"i0idhdqxqmg5oxuif8hxkzes","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5198,"modified_at":"2026-08-29T08:56:48.323000","content_hash":"b46fe2e5ac0bc72e817e7f737bb07ea872b49b62c807bd889adf9be10c873d1d"},"entries":[],"content":"# commonground-predict\n\n`commonground-predict` 0.4.1 is a deterministic Verifiers environment for\nprobabilistic masked-vote prediction over synthetic stakeholder panels. It has\nno judge model: reward is computed locally against eight masked cells per\nsnapshot.\n\nThe task is best understood as **semantic-conditioned matrix completion**.\nStatement text now causally selects a policy dimension, and latent cluster\nprofiles determine vote propensity on that dimension. The visible vote matrix\nstill carries substantial signal: a prompt-observable 5-NN comparator scores\n0.891. Results therefore do not by themselves establish policy understanding,\nhuman-preference validity, or general collective-preference reasoning.\n\nThis environment is associated with [Context Engine](https://contextengine.sh),\nwhich can structure stakeholder statements and votes into auditable preference\nmaps. A planned governed export path could let consenting individuals and\ngroups retain, license, or sell derived preference datasets. Version 0.4.1\ncontains no live participant data and no live-data exporter.\n\n## Data\n\nEach JSONL snapshot contains ordered policy statements, positional participant\nIDs, a visible vote matrix (`1` agree, `-1` disagree, `0` pass, `null` unseen),\nmasked-cell coordinates, hidden labels, planted clusters, and synthetic\nprovenance.\n\n| Split | Rows | Generator family | Statement bank |\n| --- | ---: | --- | ---: |\n| `train` | 200 | `train-random-mixture-v2` | 30 training-only statements |\n| `eval` | 100 | `heldout-archetype-threshold-v2` | 20 evaluation-only statements |\n| `ce-demo` | 1 | operator-authored synthetic fixture | adapted demo corpus |\n\nTrain and evaluation use different seeds, session IDs, policy text, and\nprofile-building families. In both synthetic families, each statement has an\nexplicit semantic dimension; changing that dimension changes its generated\nvotes. Cluster profiles, thresholding, pass behavior, and participant noise\nremain synthetic assumptions rather than estimates of people.\n\nThe CE demo fixture is also synthetic. Its immutable Context Engine source,\ntransformation boundary, hashes, and MPL-2.0 treatment are recorded in\n`NOTICE` and its byte-identical Hub-source copy `LICENSES/NOTICE.txt`.\n\n## Prompt and response contract\n\nThe prompt contains policy text, the visible matrix, and masked coordinates.\nReturn exactly one finite, non-negative probability distribution per masked\ncell:\n\n```json\n{\n  \"predictions\": {\n    \"0,5\": {\"agree\": 0.7, \"disagree\": 0.2, \"pass\": 0.1}\n  }\n}\n```\n\nThe normalized keys must exactly equal the masked cells, with no duplicates,\nand each mapping must contain\nexactly `agree`, `disagree`, and `pass` with a positive total. Values are\nnormalized before scoring. Bare hard labels are rejected. The argmax label\ndrives accuracy, with deterministic tie ordering. Completion parsing is\nbounded. Missing, extra, duplicate-normalized, or malformed output receives\nzero reward; its diagnostic Brier loss is 1.0 for non-empty tasks.\n\n## Scoring and comparators\n\n- `probability_reward`, reward weight 1.0: `1 - normalized Brier`, so calibrated\n  confidence changes the optimized objective.\n- `vote_accuracy`, metric only: exact argmax accuracy over masked cells.\n- `brier`, metric only: conventional three-class squared error divided by two,\n  bounded to `[0,1]`.\n\nExact model-free results on the bundled 100-row evaluation split:\n\n| Comparator class | Comparator | one-hot probability reward / vote_accuracy |\n| --- | --- | ---: |\n| Prompt-observable | Always agree | 0.590 |\n| Prompt-observable | Per-statement visible majority | 0.581 |\n| Prompt-observable | Nearest participant (1-NN) | 0.819 |\n| Prompt-observable | Five-neighbor vote | 0.891 |\n| Held-out-label diagnostic | Per-snapshot best constant | 0.631 |\n| Generator diagnostic | Latent cluster-pattern replay | 0.916 |\n\nThe deterministic comparators emit one-hot forecasts, for which probability\nreward equals vote accuracy. The last two rows are not floors: one reads held-out labels and one replays\nhidden generator state. The narrow gap between 5-NN and latent replay is a\ncentral limitation and should accompany any model result. Matrix-factorization,\nitem-item, spectral, text-only, matrix-only, and shuffled-text ablations remain\nrecommended additions.\n\n## Usage\n\n```bash\nuv run validate commonground-predict --runtime.type subprocess --rich false\nuv run eval commonground-predict -m MODEL --no-push\nuv run eval commonground-predict --env.taskset.split train -m MODEL --no-push\n```\n\n`split` accepts `eval`, `train`, or `ce-demo`. `data_path` or\n`COMMONGROUND_DATA_PATH` overrides the bundled split. `masked_vote_count`\ndeterministically remasks known votes; `min_cluster_count` filters rows.\nNon-synthetic custom rows must pass the separate fail-closed\n[human-data governance contract](https://github.com/doctor-gonzo/commonground-envs/blob/master/docs/human-data-governance.md).\n\nPublic answer keys make this useful for open training but unsuitable for a\ncontamination-resistant leaderboard. Use a procedurally fresh private family\nfor post-training evaluation. Native v1 and the Prime-compatible legacy adapter\nshare the same data and scoring implementation.\n","encoding":"utf-8","truncated":false,"total_bytes":5198},"status":null}