{"data":{"kind":"file","path":"README.md","version_id":"gx8w3c4vqyql2otn201ei1j8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":8389,"modified_at":"2026-08-03T14:43:36.054000","content_hash":"2c7326024bde2c61d9435115b8e9b2f841a29602beedaadc7ca1c892303ac9ca"},"entries":[],"content":"# PMPP Hard\n\nPMPP Hard is a 69-task benchmark for evaluating agents that write and optimize\nGPU kernels. The release pool contains 60 CUDA tasks, 8 Triton tasks, and 1\nCUTLASS task. Submissions must compile, produce the expected output, and satisfy\nthe configured performance gate inside isolated GPU sandboxes.\n\n## Benchmark card\n\n| Property | Release setting |\n| --- | --- |\n| Tasks | 69 total: 60 CUDA, 8 Triton, 1 CUTLASS |\n| Target architecture | NVIDIA Blackwell `sm_120`; `sm_100` is also supported |\n| Validated GPUs | RTX PRO 6000 and RTX 5090 (`sm_120`); B200 (`sm_100`) |\n| CUDA toolchain | CUDA 12.8 |\n| Evaluation framework | Verifiers 0.2.0 |\n| Performance gate | Median paired student/reference ratio at or below 1.25 |\n| Paired measurements | 5 interleaved pairs with matching output digests |\n| Source analysis | KernelGuard 0.3.2, enabled by default |\n| License | MIT |\n\nThe reported campaign uses `sm_120`; matching those timings requires equivalent\n`sm_120` hardware and the published images. The task and scoring paths also support\n`sm_100`, but scores from different GPU architectures should not be compared directly.\nLocal runs need Docker with the NVIDIA Container Toolkit. Modal examples request\n`RTX-PRO-6000` explicitly. The host driver must support the CUDA 12.8 runtime used by\nthe images.\n\n## Run PMPP Hard\n\nPMPP Hard requires Python 3.11 through 3.13, [uv](https://docs.astral.sh/uv/),\nDocker, the NVIDIA Container Toolkit, and a CUDA 12.8-compatible host driver. A\nlocal evaluation also requires a supported NVIDIA GPU and a provider API key.\n\nUse the Verifiers v1 command named `eval`. Do not use the legacy `vf-eval` or\n`vf.load_environment(...)` entrypoints; they cannot express PMPP Hard's separate\nagent runtime, clean scoring runtime, and task-specific GPU images.\n\n### Local Docker run\n\n1. Clone the repository and install its locked dependencies:\n\n   ```bash\n   git clone https://github.com/SinatrasC/pmpp-hard.git\n   cd pmpp-hard\n   uv sync --locked\n   ```\n\n2. Build the three runtime images:\n\n   ```bash\n   docker build -f docker/Dockerfile.cuda-agent -t pmpp-cuda-agent:cu128 .\n   docker build -f docker/Dockerfile.triton -t pmpp-triton:cu128 .\n   docker build -f docker/Dockerfile.cutlass -t pmpp-cutlass:12.8.1 .\n   ```\n\n3. Export the credential used by the selected configuration and run the\n   pre-spend checks:\n\n   ```bash\n   export PRIME_API_KEY=...\n   uv run python scripts/doctor.py \\\n     --config configs/smoke_k3_kimicode.toml \\\n     --base-url https://api.pinference.ai/api/v1\n   ```\n\n4. Run the one-task smoke evaluation:\n\n   ```bash\n   uv run eval @ configs/smoke_k3_kimicode.toml --no-push\n   ```\n\n5. Run a published full-pool configuration over all 69 tasks:\n\n   ```bash\n   uv run eval @ configs/campaign-2026/01-kimi-k3.toml --no-push\n   ```\n\nThe smoke configuration selects only `v2-pl-34-fused_csr_spmm_topk`. The\ncampaign configurations select all 69 tasks and map the CUDA, Triton, and\nCUTLASS lanes to their matching images. `--no-push` keeps results local; omit it\nonly when you intend to upload the run. See\n[`configs/campaign-2026/`](configs/campaign-2026/README.md) for the other reported\nmodel configurations and [`docker/README.md`](docker/README.md) for image details.\n\n### Modal run\n\nInstall the Modal dependency and authenticate:\n\n```bash\nuv sync --locked --extra modal\nuv run modal setup\n```\n\nThen validate and run the shipped one-task Modal configuration:\n\n```bash\nexport PRIME_API_KEY=...\nuv run python scripts/doctor.py \\\n  --config configs/modal_smoke_k3_kimicode.toml \\\n  --base-url https://api.pinference.ai/api/v1\nuv run eval @ configs/modal_smoke_k3_kimicode.toml --no-push\n```\n\nThe Modal smoke configuration requests an RTX PRO 6000 for the agent and clean\nscorer. For a full 69-task Modal run, publish all three runtime images to a\npublic registry and replace the local image names with immutable registry tags\nor digests. Add `--deep` to the doctor command to provision short-lived GPU\nsandboxes; this incurs provider charges.\n\n## Scoring and trust boundary\n\nThe public package contains reference material so the benchmark can be inspected\nand reproduced. That material is not placed in the evaluated agent's workspace.\nProvisioning gives the agent the task stub, contract, allowlisted build helpers,\nand, where available, a non-authoritative sanity grader. The agent never receives\nreference implementation source or authoritative grader files. CUDA sanity mode\nmay expose a source-stripped reference benchmark executable for local feedback;\nit is not the authoritative scorer.\n\nAfter the rollout, PMPP Hard copies only the submitted student source into a\nfresh scoring sandbox. The authoritative grader and reference source are\nprovisioned there, after the agent sandbox is no longer involved.\nTraces record `scored_in=clean_sandbox` for this path.\n\nA task passes only when all enabled gates pass:\n\n1. The submission builds.\n2. The authoritative correctness test passes.\n3. For a performance-gated task, five student/reference benchmark pairs run with\n   the same seed in the same scoring sandbox. Each pair must produce matching\n   output digests, and the median of `student_ms / reference_ms` must be at or\n   below the task target, normally 1.25.\n4. KernelGuard analyzes the captured source and available timing metadata. A\n   filtering source-rule verdict zeros correctness and performance; a\n   timing-anomaly verdict zeros performance. KernelGuard complements, rather\n   than replaces, compilation, correctness, digest, and paired-timing checks.\n\n## Release campaign results\n\nEach row is one accepted rollout per task (`n=1`) over the 69-task pool, so close\nscores should not be interpreted as statistically resolved rankings. Provider-void\nattempts and recovery probes are not additional accepted samples.\n\n| Model | Passed | Score | CUDA + CUTLASS | Triton | Harness, reasoning, and timeout |\n| --- | ---: | ---: | ---: | ---: | --- |\n| Kimi K3 | 49/69 | 0.710 | 42/61 | 7/8 | `kimi_code`, max, 4,500 s |\n| Claude Opus 4.8 | 43/69 | 0.623 | 35/61 | 8/8 | Codex, medium, 1,500 s |\n| GPT-5.6 Sol xhigh | 41/69 | 0.594 | 34/61 | 7/8 | Codex, xhigh, 1,500 s |\n| GPT-5.5 | 36/69 | 0.522 | 28/61 | 8/8 | Codex, medium, 1,500 s |\n| Claude Sonnet 5 | 36/69 | 0.522 | 31/61 | 5/8 | Claude Code, high, 1,500 s |\n| GLM-5.2 | 27/69 | 0.391 | 19/61 | 8/8 | Codex, medium, 1,500 s |\n| DeepSeek V4 Pro | 24/69 | 0.348 | 17/61 | 7/8 | Codex, medium, 1,500 s |\n| DeepSeek V4 Flash 0731 | 23/69 | 0.333 | 19/61 | 4/8 | Codex, high, 4,500 s |\n| GPT-5.6 Terra | 22/69 | 0.319 | 14/61 | 8/8 | Codex, medium, 1,500 s |\n| Nemotron Ultra | 18/69 | 0.261 | 10/61 | 8/8 | Codex, medium, 1,500 s |\n| GPT-5.6 Luna | 16/69 | 0.232 | 8/61 | 8/8 | Codex, medium, 1,500 s |\n\nThe table groups the 60 CUDA tasks and the single CUTLASS task into one 61-task\ncolumn. Most Codex rows used medium reasoning and a 1,500-second budget. Sol used\nxhigh reasoning. DeepSeek V4 Flash 0731 used high reasoning with a 4,500-second\nbudget, Sonnet 5 used Claude Code with high reasoning, and K3 used Kimi Code with a\n4,500-second budget. The harness and budget differences are part of the disclosed\nconfiguration rather than a claim that every row is a controlled head-to-head test.\n\nThe original Opus evaluation used Codex because Claude Code was closed source and was\nnot available as a Verifiers v1 harness. Claude Code remains closed source; the later\nSonnet 5 evaluation used the adapter implemented for this campaign.\n\nThe complete configuration recipes for all eleven reported rows are indexed under\n[`configs/campaign-2026/`](configs/campaign-2026/README.md). They pin the full task\npool, hardware architecture, harness, reasoning setting, runtime images, scoring\ncontrols, and rollout timeout. The external Claude Code adapter itself is not\nincluded in this repository, so the other ten recipes run with the bundled\ndependencies while the Sonnet recipe also requires that adapter.\n\n## Release checks\n\nThe CPU-only release suite verifies the 69-task pool, package hashes, prompt modes,\nsource isolation, scoring markers, KernelGuard integration, and negative fixtures. It\ndoes not require a GPU, Docker, provider credentials, or network access.\n\n```bash\nuv sync --locked\nuv run ruff check pmpp_hard tools scripts tests\nuv run python tests/selftest.py\nuv build\n```\n\n## License\n\nPMPP Hard is released under the [MIT License](LICENSE).\n","encoding":"utf-8","truncated":false,"total_bytes":8389},"status":null}