{"data":{"kind":"file","path":"README.md","version_id":"vcicd8mqqkeun86cyooulrto","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2573,"modified_at":"2026-08-06T10:06:21.103000","content_hash":"edf3d8467e02c68b4f9964c2f23f6f173363f218367d63640681f236a82a1480"},"entries":[],"content":"# web3-anchor-bench\n\nPublic dev split of the **Web3 anchor-layer benchmark**: crypto/Web3 market-analysis\nquestions where a frozen snapshot of real data (on-chain metrics, prices, funding\nrates, prediction-market odds, token security scans, …) is embedded in the prompt,\nand the answer is deterministically derivable from that data alone.\n\nThe design goal is **faithfulness under grounding**: the model must answer only\nfrom the given data and must not introduce numbers that are not present in the\nquestion. Grading is fully deterministic — no LLM judge in the loop.\n\n## What's inside\n\n- **33 single-turn cases** (22 Chinese + 11 English), 10+ domains: BTC cycle\n  indicators, technicals, DeFi yields, on-chain flows, prediction markets,\n  perps funding, token safety scans, macro, sentiment, and more.\n- Answers use a **fixed three-section format** (Conclusion / Key Data / Reasoning)\n  so the fabrication signal only scans the model-declared key-data section and\n  domain thresholds in free-form reasoning don't false-positive.\n- **Deterministic rubric** (ported byte-identically from the leaderboard pipeline):\n  - `anchor_pass` (reward): required conclusion concepts hit AND required values present\n  - `conclusion_match`, `numeric_match`, `format_compliance` (metrics)\n  - `ungrounded_numbers` (metric): count of key-data numbers absent from the snapshot\n\n## Important: this is the DEV split\n\nQuestions here are published **with answers** for development and reproduction.\nReal ranking runs on a **private held-out split** (canary-watermarked, not\npublished) with identical domain/difficulty stratification. Scores on this\npublic split do not transfer to the held-out ranking.\n\n## Usage\n\n```bash\nprime env install web3-anchor-bench\nvf-eval web3-anchor-bench -m <model> -b <openai-compatible-base-url> -k <API_KEY_VAR> -n 33 -r 1\n```\n\nEnvironment args:\n\n```bash\n# Chinese-only or English-only subset\nvf-eval web3-anchor-bench -a '{\"lang\": \"zh\"}' ...\nvf-eval web3-anchor-bench -a '{\"lang\": \"en\"}' ...\n```\n\nReference results (fixed format, this public split, 2026-08): production-tier\nmodels score 31–33 / 33; the split is a floor check, not a discriminator —\nopen-ended faithfulness discrimination lives in the judged layer of the full\nbenchmark.\n\n## Methodology & leaderboard\n\nFull methodology (three-layer design, judge panel for the open-ended layer,\nanti-contamination protocol, caveats) and the current leaderboard are published\nin the source repository. Licensed MIT; adversarial safety scenarios derive from\nthe Minara set (MIT, attribution in LICENSE).\n","encoding":"utf-8","truncated":false,"total_bytes":2573},"status":null}