{"data":{"kind":"file","path":"README.md","version_id":"kcthdwk40zxf3kaw04y2ovjm","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":1868,"modified_at":"2026-09-13T04:06:06.767000","content_hash":"f319a86b60a605d21b0513f36eead3bf5b0ea3a8d9edafa43b4f5edb8aff6a78"},"entries":[],"content":"# humaneval\n\nHumanEval code generation using the exact instruction from [OpenAI simple-evals](https://github.com/openai/simple-evals/blob/main/humaneval_eval.py), also used by [Inspect Evals](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/humaneval/humaneval.py) and Prime's community HumanEval environment:\n\n> Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n\nThe instruction and the official `openai/openai_humaneval` function prompt are sent as one user message. There is no system prompt, synthetic demonstration, or task-specific chain-of-thought instruction.\n\n## Taskset\n\n- **Source:** [openai/openai_humaneval](https://huggingface.co/datasets/openai/openai_humaneval)\n- **Size:** 164 test tasks\n- **Prompting:** zero-shot\n- **Reward:** 1.0 iff the generated function passes every test\n\nScoring runs the official HumanEval tests inside the selected Verifiers runtime. The candidate runs in a spawned worker that receives the solution and function calls, but never the test source.\n\nThe benchmark's shot count is fixed at zero. Use `-r` to control independent samples per task, as in OpenAI simple-evals' `num_samples_per_task`. Model-native reasoning remains configurable through the evaluation sampling options when supported by the selected model.\n\n```bash\nuv run eval humaneval \\\n  -m deepseek/deepseek-v4-flash \\\n  --env.agent.harness.id null \\\n  -n 3 -r 1 --no-rich -v --no-push\n```\n\n## Security\n\nModel-generated Python executes inside the selected Verifiers runtime. The HumanEval reliability guard is not a security sandbox; use a Docker or Prime runtime when evaluating models you do not trust.\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n","encoding":"utf-8","truncated":false,"total_bytes":1868},"status":null}