{"data":{"kind":"file","path":"README.md","version_id":"q8yx605uy0aqc3mlbgdb6ajz","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2943,"modified_at":"2026-09-13T04:06:06.924000","content_hash":"efe79c7ebccfaf9aa8b2ca5151c25a2580cf2b9337f5e7aa828742d47dc95c60"},"entries":[],"content":"# triviaqa\n\nTriviaQA knowledge questions with configurable few-shot prompting, evidence, and alias-aware exact-match scoring.\n\n## Taskset\n\n- **Source:** [`mandarjoshi/trivia_qa`](https://huggingface.co/datasets/mandarjoshi/trivia_qa), `rc.wikipedia.nocontext` configuration\n- **Evaluation split:** `validation` (7,993 tasks)\n- **Few-shot split:** `train` (61,888 examples)\n- **Metric:** normalized exact match against every answer alias; generation is cut at the first newline, period, or comma\n\n`rc.wikipedia.nocontext` is the commonly used Wikipedia subset for closed-book TriviaQA. It keeps only questions with Wikipedia evidence while omitting those documents from the dataset rows and prompts. The labeled `test` split is not public, so the benchmark evaluates on `validation`.\n\nBy default, each evaluation question receives five Q/A demonstrations sampled from the training split. Demonstrations use each training row's canonical `answer.value`. A single seeded random generator advances through the validation set, so each question gets a deterministic sample and repeated loads produce identical prompts. Demonstrations and the evaluation question are placed together in one prompt:\n\n```text\nQuestion: <training question>\nAnswer: <canonical training answer>\n\n...\n\nQuestion: <evaluation question>\nAnswer:\n```\n\n## Configuration\n\n| Option | Default | Effect |\n| --- | ---: | --- |\n| `--env.taskset.shots` | `5` | Number of training demonstrations; `0` also skips loading the few-shot split. |\n| `--env.taskset.dataset-config` | `rc.wikipedia.nocontext` | Hugging Face TriviaQA configuration. |\n| `--env.taskset.fewshot-split` | `train` | Split sampled for demonstrations. |\n| `--env.taskset.eval-split` | `validation` | Split used for scored tasks. |\n| `--env.taskset.num-examples` | `7993` | Maximum number of evaluation rows loaded from the split. |\n| `--env.taskset.include-evidence` | `false` | Adds Wikipedia/search evidence to every demonstration and question. When enabled with a `.nocontext` dataset config, the suffix is removed to load its context-bearing sibling. |\n| `--env.taskset.metric` | `normalized_alias_exact_match` | TriviaQA-normalized alias exact match. `alias_exact_match` selects case-sensitive raw-alias exact match. |\n| `--env.taskset.fewshot-seed` | `1234` | Seed for deterministic demonstration sampling. |\n\n`normalized_alias_exact_match` uses the dataset's `normalized_aliases` and the official TriviaQA normalization rules: lowercase text, replace underscores, remove punctuation and articles, and collapse whitespace.\n\n## Evaluation\n\n```bash\nuv run --no-sync eval triviaqa -n 3 -r 1 --no-rich -v --no-push \\\n  --env.agent.harness.id null --sampling.temperature 0 --sampling.max-tokens 256\n```\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-07-20: Initial v1 taskset with configurable five-shot `rc.wikipedia.nocontext` defaults.\n","encoding":"utf-8","truncated":false,"total_bytes":2943},"status":null}