{"data":{"kind":"file","path":"README.md","version_id":"c10774bhach3de25df9laqw6","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2540,"modified_at":"2026-09-13T04:06:06.963000","content_hash":"701f9e1f489362793cec591d2c42f034058333b5e4994239c3a23e90cba4674f"},"entries":[],"content":"# pinchbench\n\nPinchBench personal-assistant and knowledge-work tasks run by a CLI-agent harness in a sandbox. Tasks are scored host-side with the original task-embedded automated checks and an LLM judge (a weighted hybrid by default).\n\n## Taskset\n\n- **Source:** [`pinchbench/skill`](https://github.com/pinchbench/skill) (task markdown copied verbatim, `TASK_TEMPLATE.md` excluded)\n- **Size:** 23 tasks\n\n## Security\n\nAutomated grading runs untrusted, task-embedded Python via `exec()` in the eval (host) process — **outside** the agent sandbox. It is scoped to a fresh namespace fed only the parsed transcript and the downloaded workspace path, but it is still arbitrary code running with the evaluator's credentials. Only run task files you trust.\n\nSolver networking is default-deny. The stock, conference-research, and image-generation tasks retain public access because their prompts require live services; the weather task is limited to `wttr.in`. The other 19 tasks are self-contained or make web use optional.\n\n## Changelog\n\n- 2026-08-31: Yield task records on demand so bounded evaluations construct only the requested prefix.\n- 2026-07-31: Renamed the task's solve-phase timeout override from `harness` to `agent` for `verifiers>=0.2.2.dev65`, preserving the per-task budget.\n- 2026-07-29: Removed the redundant hatchling `force-include` of package data dirs (already shipped via `packages`); newer hatchling errors on the duplicate archive paths.\n- 2026-07-10: Ported to the task-centric verifiers API: rewards and lifecycle hooks live on the\n  `Task` (a `TaskData` row + behavior split), and task-facing config knobs (judges, tool/user\n  placement, scoring parameters) moved from `--env.taskset.*` to `--env.taskset.task.*`. Requires\n  `verifiers>=0.2.0` and Python `>=3.11`.\n  - The agent is now the v1 CLI-agent harness (`default`/`rlm`/`bash`/...) instead of the bundled\n    OpenClaw CLI, mirroring `terminal_bench_2` / `swebench_verified` (the v1 idiom: the\n    harness is the swappable agent). The grading transcript is reconstructed from the rollout;\n    the embedded `grade(...)` functions already accept multiple agent tool-call conventions.\n    Consequently the v0 `openclaw_version` and `setup_parallelism` knobs are dropped, and the\n    single multi-session task (`task_22_second_brain`) runs as one turn (no cross-session recall).\n  - The upstream default judge model string is preserved (its leading `openrouter/` prefix\n    dropped, as v0 did before sending), now on the Prime inference endpoint via `JudgeConfig`.\n","encoding":"utf-8","truncated":false,"total_bytes":2540},"status":null}