{"data":{"kind":"file","path":"README.md","version_id":"vcuqem9s1r5x0xsh2lfwtswa","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":4273,"modified_at":"2026-07-28T00:52:17.203000","content_hash":"45bf37a7b8f0d3603b416122ca936d2d4b07e85c6fdf1eaffdc54284b6b42fea"},"entries":[],"content":"# prime-forecast\n\nAgentic forecasting RLVR environment for [verifiers](https://github.com/PrimeIntellect-ai/verifiers) / Prime Intellect Hosted Training.\n\nA multi-turn agent receives a resolved Polymarket binary question with a historical cutoff date, researches it with live cutoff-safe tools (Firecrawl/Brave/Tavily/AgentCore/Exa web search, yfinance, FRED, DBnomics, Wikipedia-as-of-cutoff, trend analysis), maintains a BLF-style structured belief state, and submits `P(YES)`. Reward is positive-shifted Brier against the real outcome:\n\n```\nreward = 1 - (p - y)^2        (0.55 soft no-submit)\n```\n\nMissing submission scores 0.55 (below always-0.5) so stalling is not a safe reward; optional protocol bonus defaults to off. Cutoff-safe Polymarket crowd tools (`include_market_tools=True` by default) expose price-at-cutoff, price history, and dataset volume; `market_brier` remains an eval metric. Use the zero-advantage pre-batch filter to drop collapsed crowd-copy groups.\n\n## Leak safety\n\n1. Provider date filter (Exa `endPublishedDate`) + client-side publish-date drop\n2. Prediction-market domain blocklist (polymarket, kalshi, manifold, metaculus, predictit)\n3. Heuristic post-cutoff date/outcome-language filter on all retrieved text\n4. Bedrock LLM leak filter (Claude Haiku by default) with per-result KEEP/DROP votes\n5. All time-series/Wikipedia tools clamp end dates to the cutoff; model-supplied end dates can only tighten, never extend\n\n## Required Environment Variables\n\n| Variable | Required | Purpose |\n|---|---|---|\n| `PF_SEARCH_BACKEND` | recommended | `firecrawl` \\| `brave` \\| `tavily` \\| `agentcore` \\| `exa` \\| `none` |\n| `FIRECRAWL_API_KEY` | for firecrawl | Firecrawl Search API (tbs date-range cutoff; 1 credit/search) |\n| `BRAVE_API_KEY` | for brave | Brave Search API (cheap; leak filter supplies cutoff) |\n| `TAVILY_API_KEY` | for tavily | Tavily Search (free tier; local eval / small smokes) |\n| `AGENTCORE_GATEWAY_URL` | for agentcore | MCP gateway URL from `scripts/setup_agentcore_search.py` |\n| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` / `AWS_PROFILE` | for agentcore | SigV4 for Gateway MCP (`us-east-1`) + Bedrock leak filter |\n| `AWS_BEARER_TOKEN_BEDROCK` | optional | Bedrock Runtime only (leak filter); **not** enough for AgentCore |\n| `EXA_API_KEY` | for exa | Exa web search |\n| `FRED_API_KEY` | optional | `fetch_fred_series` tool |\n| `PF_LLM_FILTER` | optional | force LLM filter on/off (`1`/`0`); auto-detects AWS creds by default |\n| `PF_FILTER_MODEL` | optional | leak-filter model (default Claude Haiku 4.5 — benchmarked best; `qwen.qwen3-32b-v1:0` is the validated ~7x-cheaper fallback: zero false-KEEPs, ~12% over-drop) |\n| `PF_SEARCH_CACHE` | optional | `0` disables the search cache (default on) |\n| `PF_SEARCH_CACHE_DIR` | optional | cache/corpus location (default: system temp) |\n\n## Search cache\n\n`web_search` results are cached **post-leak-filter**, keyed on (backend, cutoff,\nnormalized query). A hit costs zero search-API credits and zero Bedrock filter\ncalls — GRPO groups re-searching the same phrasing get identical context. The\ncache directory doubles as a publishable corpus: each entry stores the query,\ncutoff, and the filtered results the agent was shown.\n\n## Environment args\n\n| Arg | Default | Description |\n|---|---|---|\n| `dataset_path` | packaged data | JSONL file or dir with `{split}.jsonl` |\n| `split` / `eval_split` | `train` / `val` | dataset splits |\n| `num_examples` / `num_eval_examples` | `-1` | truncate splits |\n| `max_turns` | `8` | BLF T_max |\n| `max_web_searches` | `2` | cap live `web_search` calls per rollout |\n| `include_market_tools` | `true` | expose cutoff-safe Polymarket price/history/volume tools |\n| `crowd_copy_penalty` | `0.20` | subtract from reward when `|p - crowd| ≤ crowd_copy_eps` |\n| `crowd_copy_eps` | `0.02` | absolute distance to `price_at_cutoff` treated as a copy |\n| `protocol_bonus_weight` | `0.0` | optional BLF shaping (off by default) |\n\n## Dataset\n\nBuilt from the Polymarket Gamma + CLOB APIs by `scripts/build_dataset.py` (repo root): resolved binary YES/NO markets, volume/price filters, per-market cutoff 7 days before resolution, temporal train/val/test splits. Run with `--install` to copy splits into the package data directory.\n","encoding":"utf-8","truncated":false,"total_bytes":4273},"status":null}