{"data":{"kind":"file","path":"README.md","version_id":"we8s86dh8wi1bqvhuck8rvpk","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1579,"modified_at":"2026-05-10T23:51:10.797000","content_hash":"a3863d3e2e4332b72d864b44cca952f06e8f361e28a3c1a79b7a7a28b5772dce"},"entries":[],"content":"# TritonBench Verifiers Environment\n\nKernelBench-style layout for **Triton** kernels + **Modal** GPU evaluation. Dataset: `anna4142/triton-bench-verifiers`.\n\n## Layout (matches KernelBench)\n\n| Path | Purpose |\n|------|---------|\n| `tritonbench.py` | `load_environment`, dataset builder, `TritonBenchRubric` |\n| `tritonbench_env/modal_sandbox.py` | Modal image + `modal_eval_triton` / baseline |\n| `tritonbench_env/triton_harness.py` | Reference vs candidate exec, timing (µs) |\n| `tritonbench_env/baseline_cache.py` | Local JSON cache for reference timings |\n| `tritonbench_env/parser.py` | Fenced code extraction |\n| `tritonbench_env/score.py` | Metric extractors |\n| `tritonbench_env/sandbox.py` | Prime sandbox stub (same pattern as KernelBench) |\n| `prompts/` | `prompt_constructor.py`, `hardware/gpu_specs.py`, minimal Triton examples |\n\n## Metrics\n\nSame names as KernelBench: `gmsr_correct`, `fast_0`, `fast_1`, `fast_2`, `speedup`. Default `reward_metric`: `gmsr_correct`.\n\n## Quickstart\n\n```bash\nuv pip install -e .\n# Modal:\nmodal setup\n```\n\n```python\nfrom verifiers import load_environment\nenv = load_environment(\"tritonbench\", max_samples=10)\n```\n\nLocal CPU dev (no Modal): `load_environment(..., use_modal_eval=False)` — needs CUDA for meaningful scores.\n\n## Eval CLI\n\n```bash\nuv run vf-eval tritonbench -p openai -m gpt-4o-mini -n 1 -r 1 -t 2048 \\\n  -a '{\"max_samples\": 5, \"use_modal_eval\": true}'\n```\n\n## Note\n\nThe internal package is named `tritonbench_env` (not `utils`) so it does not clash with other environments’ `utils` when multiple envs are installed.\n","encoding":"utf-8","truncated":false,"total_bytes":1579},"status":null}