{"data":{"kind":"file","path":"README.md","version_id":"qpmp0bn3svyhzctffe1c094w","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2607,"modified_at":"2026-07-03T15:10:06.435000","content_hash":"4cc4a9f5a5949858e936faf1a9019b9216741d2434ac9e4cb0152e07681701ae"},"entries":[],"content":"# football-culture-reasoning\n\nExpert-graded evaluation of LLM reasoning about **football (soccer) fandom culture**: subcultural concepts, non-Western specificity (Japan/J.League, South America, Asia), macro-sociological context, and stereotype avoidance.\n\n### Task\n\nSingle-turn. The model is shown a topic and a realistic, fluent, model-generated answer (in Japanese) and must reproduce a domain expert's judgment:\n\n- **verdict**: `good` / `borderline` / `bad`\n- **tags**: error taxonomy (11 tags): `WC` West-centrism, `OLD` outdated framing, `CONF` concept conflation, `SIMP` oversimplification, `FACT` factual error, `STER` stereotype, `MACRO` missing macro-social structure, `DISM` dismissiveness, `NORM` normative assertion, `DEBUNK` repeating debunked narratives, `GOOD` sound.\n\n### Reward\n\n`0.6 * verdict_exact + 0.35 * tag_jaccard + 0.05 * format`\n\n### Why this is hard\n\nThe candidate answers are fluent and confident; the failures are cultural, not linguistic. Baseline finding: a blind frontier model reproduces the expert's coarse verdicts on 18/20 items, but systematically fails the calibration cases (expert-\"borderline\" items it flattens to \"bad\") and cannot reproduce the expert rationales — counterexamples (Brighton–Palace as a non-geographic derby), currency corrections (\"corporate J.League\" is a ~2000s framing; support culture fused with Japanese baseball cheering traditions), and causal fixes (Hillsborough: hooligan decline as a byproduct of ticket-price/stadium changes, not \"strengthened countermeasures\").\n\nCommon failure modes this eval punishes: West-centric defaults, decades-stale framings of non-Western fandom, normative moralizing (\"politics should stay out of sport\") instead of explanation, and dismissal of sociological structure (class, migration, political economy).\n\n### Ground truth\n\nSingle-expert, rationale-backed, second-pass labels by **Kei Akiyoshi** — football journalist, commentator (DAZN EFL / U-NEXT FA Cup) and author based in Japan, specializing in fandom sociology. Each item also carries the expert's written correction (`info.expert_fix`), usable as reference text for judge-based generative evaluation.\n\n### Usage\n\n```bash\nprime env install <owner>/football-culture-reasoning\nuv run vf-eval football-culture-reasoning -m <model>\n```\n\n### Extensibility\n\nv1 = 20 items (proof set). The rubric and pipeline (AI drafts candidate answers → expert adjudicates with tags + written rationale) are domain-portable: tactics discourse, scouting narratives, other fandom subdomains, and other non-Western cultural domains. Contact: kei@japanesethe72.com\n","encoding":"utf-8","truncated":false,"total_bytes":2607},"status":null}