{"data":{"kind":"file","path":"README.md","version_id":"buw66r10lpozdbcok3sl1vpb","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2058,"modified_at":"2026-04-23T15:19:43.027000","content_hash":"fa31a219de601a694ad4493980675f446c4f67543318c507c884f54fc480eeff"},"entries":[],"content":"# InfraResolutionBench\n\nPrime `verifiers` environment module for InfraResolutionBench in this repo.\n\n## What it does\n\n- loads the benchmark dataset directly from this repository\n- packages gold and synthetic case JSON files inside the environment wheel for Hub and hosted execution\n- builds Prime-compatible prompts from the existing benchmark contract\n- scores outputs with deterministic exact, consistency, and rubric logic\n- returns a single composite reward weighted toward exact correctness: `0.7 exact / 0.2 consistency / 0.1 rubric`\n\n## Supported environment arguments\n\n- `case_source=\"gold\" | \"synthetic\" | \"all\"`\n- `prompt_mode=\"packet\" | \"tools\"`\n- `limit=<int | None>`\n\n## Current prompt modes\n\n- `packet`\n  Sends the full case packet directly in the prompt.\n- `tools`\n  Uses live read-only tool calling through `vf.ToolEnv`. The model receives the case ID in the prompt and can call record-access tools until it is ready to return the final JSON answer.\n\n## Local setup\n\nInstall the Prime CLI and log in:\n\n```bash\nuv tool install prime\nprime login\n```\n\nIf you prefer API-key auth:\n\n```bash\nprime config set-api-key\nprime config set-team-id  # optional, only if you want team-scoped access\n```\n\nInstall the environment from this repo:\n\n```bash\nprime env install infraresolutionbench -p ./environments\n```\n\nRun a local eval:\n\n```bash\nprime eval run infraresolutionbench -m gpt-5-nano\n```\n\nRun a packet-mode eval on only the handwritten seed set:\n\n```bash\nprime eval run infraresolutionbench -m gpt-5-nano --env-args '{\"case_source\":\"gold\",\"prompt_mode\":\"packet\",\"limit\":15}'\n```\n\nPublish to the Environments Hub:\n\n```bash\nprime env push infraresolutionbench -p ./environments\n```\n\n## Required environment variables\n\nThis environment does not require any extra service keys of its own for scoring.\n\nFor Prime account access and hosted evaluation you still need Prime authentication through:\n\n- `prime login`, or\n- `PRIME_API_KEY` / `prime config set-api-key`\n\nIf you use a team workspace, also set `PRIME_TEAM_ID` or run `prime config set-team-id`.\n","encoding":"utf-8","truncated":false,"total_bytes":2058},"status":null}