{"data":{"kind":"file","path":"README.md","version_id":"yisfaaeinci0f3ocbzrgm8ws","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2513,"modified_at":"2026-09-04T11:59:16.787000","content_hash":"a585375ae761c231165b319e6d2fc6075385d95c2fd7c6c2702ee50536e85a23"},"entries":[],"content":"# triviaqa\n\nTriviaQA open-domain trivia question answering evaluation environment for the\n[Prime Intellect Environments Hub](https://app.primeintellect.ai/dashboard/environments),\nbuilt on the [verifiers](https://github.com/PrimeIntellect-ai/verifiers) framework.\n\nSource implementation: https://github.com/PrimeIntellect-ai/community-environments (this push; authored by @ux-xd).\n\n## Description\n\nImplements the TriviaQA benchmark (Joshi et al., 2017) on the `rc.nocontext`\nconfiguration of [mandarjoshi/trivia_qa](https://huggingface.co/datasets/mandarjoshi/trivia_qa).\nEach rollout is a single-turn, closed-book trivia question; scoring is the\nofficial TriviaQA exact match: the model's answer is normalized (lowercase,\npunctuation stripped, articles removed, whitespace collapsed) and compared\nagainst the question's full normalized alias set provided by the dataset.\n\n## Dataset\n\n- Split `train` of `rc.nocontext` supplies the training dataset (default 1000 examples).\n- Split `validation` of `rc.nocontext` supplies the evaluation dataset (default 500 examples).\n- Both splits are shuffled with a fixed seed (default 0), so task sets are\n  deterministic and reproducible for a given `(num_train, num_eval, seed)`.\n\nEach dataset row carries:\n\n- `question` — instructions + the trivia question\n- `answer` — the canonical answer string (`answer.value`)\n- `info` — JSON with `aliases` and the dataset's precomputed `normalized_aliases`\n\n## Reward functions\n\n| Function | Weight | Description |\n| --- | --- | --- |\n| `exact_match` | 1.0 | 1.0 iff the normalized extracted answer matches any normalized alias (official EM) |\n| `answer_line_given` | 0.0 (metric) | Whether the response followed the required `Answer: <answer>` format |\n\nAnswer extraction prefers the text after the last `Answer:` marker and falls\nback to the last non-empty line of the response. Scoring is fully\ndeterministic — no LLM judge, no network calls at reward time.\n\n## Dependencies\n\n- `verifiers>=0.1.11`\n- `datasets>=3.0.0`\n\nNo API keys, secrets, or extra setup are required: installing the environment\nand running an evaluation works out of the box.\n\n## Usage\n\n```bash\nprime env install <owner>/triviaqa\n```\n\nEvaluate with the Prime CLI:\n\n```bash\nuv run vf-eval triviaqa -m gpt-4.1-mini -n 20 -r 5\n```\n\nConfiguration knobs (all optional):\n\n- `num_train` — training split size (default 1000, `None` = full split)\n- `num_eval` — evaluation split size (default 500, `None` = full split)\n- `seed` — shuffle seed (default 0)\n","encoding":"utf-8","truncated":false,"total_bytes":2513},"status":null}