{"data":{"kind":"file","path":"README.md","version_id":"xrlmwqy9dcdmgnd6jh89lf5o","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8979,"modified_at":"2026-09-09T10:17:12.469000","content_hash":"11f017b4fd8e90abb206671ecc817533911be0abbe3c4b5b070bdada7ca86a1e"},"entries":[],"content":"# calibrated-forecasting\n\nProbability forecasting on real events that have since resolved, under a hard\ninformation cutoff, scored with strictly proper scoring rules against a liquid\nprediction market.\n\nThe model sees a question and an \"as of\" timestamp, and must end its reply with:\n\n```\nPROBABILITY: 0.62\n```\n\nIt is scored on what actually happened.\n\n**16,705 items · 7,163 markets · 32 days of resolutions (2026-08-09 → 09-09) ·\nhorizons of 24 / 72 / 168 hours.**\n\n## Why the grader is hard to game\n\nMost eval graders are either an exact-match string or a judge model. The first\nis brittle, the second can be talked into things. This one is neither.\n\n**The scoring rule is strictly proper.** Reward is `1 - (p - outcome)²`. The\ndefining property of a strictly proper rule is that expected reward is maximised\n*only* by reporting your true belief — misreporting is worse in expectation, for\nevery possible true belief. There is no verbal strategy, no hedge, and no\nformatting trick that beats being calibrated. `test_rubric.py` checks this\nnumerically rather than asserting it.\n\n**Ground truth is a settled outcome**, taken from the market's own on-chain\nresolution, not from a model's opinion.\n\n**Degenerate strategies visibly lose.** Measured on the shipped data:\n\n| Strategy | Mean reward |\n|---|---|\n| always 0.95 (favourite spam) | 0.4731 |\n| always 0.05 (longshot spam) | 0.6219 |\n| always 0.5 (hedge everything) | 0.7500 |\n| **the market price at cutoff** | **0.8291** |\n| perfect oracle | 1.0000 |\n\nRead the band, not the absolute number: **0.7500 is the score for having no\nopinion at all**, and 0.8291 is a liquid market with real money behind it.\nA model below 0.75 is worse than useless. The `market_skill` metric rescales\nthis so 0 means \"matched the market\" and positive means genuinely beat it.\n\nOn the non-sports subset the market is a *harder* opponent still (0.8921), which\nmakes that slice the better discriminator between models.\n\n## Recall is not forecasting\n\nIf an event resolved before a model's training cutoff, the model may simply\nremember how it turned out. The benchmark would then be measuring memory while\nreporting a forecasting score. This is the single most common way a forecasting\neval quietly becomes fiction.\n\n`contamination_probe(cutoff_iso)` splits the data by resolution date and reports\nboth halves, along with the market's Brier on each so you can confirm the halves\nare of comparable difficulty before blaming a model:\n\n```python\nfrom calibrated_forecasting import contamination_probe\ncontamination_probe(\"2026-08-25T00:00:00Z\")\n```\n\nA model that genuinely forecasts scores about the same on both halves. A model\nscoring materially better on events that resolved before its cutoff is recalling.\n\n## Metrics\n\nOnly `brier_reward` carries weight. The rest are reported as diagnostics.\n\n| Metric | Weight | What it tells you |\n|---|---|---|\n| `brier_reward` | 1.0 | The score. Strictly proper. |\n| `log_reward` | 0.0 | Second proper rule, far harsher on confident errors. A model that is well-behaved but overconfident looks fine on Brier and bad here. |\n| `market_skill` | 0.0 | Brier skill vs the market at cutoff. >0 beats the market. |\n| `format_ok` | 0.0 | Share of replies that parsed. If this drifts below 1.0, the headline is partly measuring formatting. |\n| `confidence` | 0.0 | Mean \\|p−0.5\\|×2. Exposes the hedging strategy: a model that answers 0.5 to everything shows ~0.0 here while still scoring 0.75. |\n\n`market_skill` is deliberately kept out of the training signal. The market is\nclose to efficient, so optimising against it directly rewards noise around a\nnear-unbeatable baseline.\n\n## Usage\n\n```python\nimport verifiers as vf\n\n# Everything.\nenv = vf.load_environment(\"calibrated-forecasting\")\n\n# General-knowledge slice: no match predictions, no near-decided questions.\nenv = vf.load_environment(\n    \"calibrated-forecasting\",\n    exclude_tags=[\"Sports\", \"Esports\", \"Games\"],\n    exclude_near_certain=0.05,\n)\n```\n\n| Argument | Default | Effect |\n|---|---|---|\n| `horizons` | all | Keep only these forecast horizons, in hours (24, 72, 168). |\n| `min_volume` | 0 | Drop markets thinner than this, in USD. |\n| `exclude_near_certain` | 0.0 | Drop items whose market price is within this of 0 or 1. `0.05` strips near-decided questions, which are free marks that flatter every model equally. |\n| `include_tags` | none | Keep only items carrying one of these tags (case-insensitive). |\n| `exclude_tags` | none | Drop items carrying any of these tags. |\n| `max_items` | None | Cap dataset size (chronological, so a truncated set is a contiguous period). |\n\nRun the grader checks with no API key and no model:\n\n```bash\npython test_rubric.py\n```\n\n## Data\n\nEach row carries the question, the cutoff, the resolved outcome, the market's\nlast traded price at that cutoff, and Polymarket's own category tags.\n\nComposition by tag (rows may carry several):\n\n| Tag | Rows | Share |\n|---|---|---|\n| Sports | 10,926 | 65.4% |\n| Politics | 1,728 | 10.3% |\n| Crypto | 1,644 | 9.8% |\n| Weather | 1,167 | 7.0% |\n| Elections | 709 | 4.2% |\n| Tech | 341 | 2.0% |\n| Economy | 205 | 1.2% |\n\nExcluding `Sports`, `Esports` and `Games` leaves **5,764 items**. Tags are\nPolymarket's own labels rather than keyword guesses over the question text —\nwhich matters, because text heuristics misclassify esports and exact-score\nmarkets as general questions.\n\nRebuild or extend with `harvest.py`:\n\n```bash\npython harvest.py --pages 30 --min-volume 15000 --horizons 24 72 168   # recent\npython harvest.py --back-days 30 --pages 8 --min-volume 15000          # backfill\n```\n\nRuns are cumulative — each `(market, horizon)` pair is written once, so running\nit on a schedule grows the archive.\n\n**Why it must be run on a schedule.** The CLOB `prices-history` endpoint serves\nonly a trailing window. Measured directly by sampling markets at increasing age:\n1, 2, 3 and 4 weeks old return full hourly history; **6, 8 and 12 weeks old\nreturn an empty array**. Beyond roughly five weeks the price at cutoff is not\nrecoverable from the public API at any later date, so the baseline this\nbenchmark scores against exists only if it is captured inside that window.\n\nTwo scan modes exist because they fail in different places. The recent scan\norders by close time and pages by offset, but gamma stops responding near offset\n2100 — about three days back. The backfill walks one-day `end_date` windows\ninstead, which never hits that ceiling and reaches as far as history survives.\nThe recent scan still catches markets whose `endDate` is a far-future\nplaceholder, which the window walk misses, so both are kept.\n\nOnly clean binary settlements are kept — `outcomePrices` exactly `{\"0\",\"1\"}` with\n`Yes`/`No` outcomes. Ambiguous and multi-outcome markets are dropped rather than\nguessed at (about 85% of scanned markets). Prices are never interpolated forward\npast the cutoff, and a cutoff whose last trade is older than `--max-stale-min` is\nrejected rather than carried.\n\nItems are also required to have existed at their own cutoff. That rule was\nwritten for correctness — a market created after the cutoff has no forecast to\nmake — and it turned out to matter far more than expected: it excludes every\n\"Bitcoin Up or Down\" five-minute market automatically, and those are roughly\ntwo thirds of all recently-resolved markets. Without it the benchmark would be\nmostly coinflips.\n\n## Known limitations\n\nStated because they change how the number should be read.\n\n- **Sports is 65% of rows.** A month of data did not change this; it is simply\n  what Polymarket's resolved binary markets are. Use `exclude_tags` for a\n  general-knowledge subset, at the cost of dropping to 5,764 items.\n- **32 days of resolutions.** The archive grows daily but cannot be extended\n  backwards, for the retention reason above.\n- **The scoring band is compressed** (0.75 to ~0.83). Inherent to Brier on\n  binary events, and why `market_skill` exists. Small differences in the\n  headline number are not meaningful without the item count.\n- **The market baseline is a strong reference, not ground truth about\n  difficulty.** On the uncertain subset (price 0.05–0.95) the market's own Brier\n  is 0.2223 — those events are genuinely close to coinflips.\n\n## A note on the parser\n\nThe parser rejects rather than guesses. An unparseable reply scores 0.0, not\n0.5 — otherwise a model could raise its average by declining the items it finds\nhard.\n\nOne case is worth calling out because it is a live example of the failure mode\nthis benchmark is built to resist. An early version read a bare number above 1\nas a percentage, so `PROBABILITY: 1.7` became `0.017`. That silently converted a\nmalformed reply into a *confident* forecast, which then scored ~1.0 on any event\nthat resolved NO — free reward for garbage output. The ambiguous interval (1, 2)\nis now rejected outright, while unambiguous percentages like `62` and `62.5`\nstill parse. `test_rubric.py` covers the regression.\n","encoding":"utf-8","truncated":false,"total_bytes":8979},"status":null}