{"data":{"kind":"file","path":"README.md","version_id":"wkfrrqas3lknsry8wxph50pi","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3231,"modified_at":"2026-09-10T10:57:25.715000","content_hash":"1d06617ebccaf7c60640f384f1b6b95208357590ecdb3c622db9fd74804a7c94"},"entries":[],"content":"# zenml-bench (verifiers v1 taskset)\n\n### Overview\n- **Environment ID**: `zenml-bench`\n- **Short description**: coding agents build, fix, configure and do research loops with real ZenML 0.96.4; every task is graded on the state ZenML recorded (runs, step statuses, artifacts, compiled configuration, model version stages), never on the agent's source text.\n- **Tags**: zenml, mlops, harbor, agentic, terminal, train, eval\n\n### Datasets\n- **Primary dataset**: the 18 Harbor tasks in https://github.com/zenml-io/zenml-bench (`tasks/`), listed in the repo's `registry.json`; Hub id `zenml/zenml-bench@0.1.0` once public.\n- **Split sizes**: 18 eval tasks (v0.1). Generated training instances (`scripts/generate_instances.py`) are a separate, larger pool and are not in this taskset.\n\n### Task\n- **Type**: multi-turn, sandboxed terminal (Harbor task format: `instruction.md`, `environment/Dockerfile` on a pinned base image, hidden `tests/`).\n- **Rubric**: each task's own frozen verifier writes `/logs/verifier/reward.json` with a binary `reward` and, for some tasks, extra metrics (`gap_closed`, `hidden_log_loss`, `agent_runs`, ...) that land in the trace's `metrics`. Every task passed the four checks: reference solution 5/5, doing nothing 0, every listed shortcut 0, at least one alternative solution 1.\n\n### Images\nverifiers never builds a task's `environment/Dockerfile`; it only pulls or reuses an image by name. Every zenml-bench task is Dockerfile-only (the Dockerfile copies the project in and seeds the ZenML store), so this taskset sets `ignore_dockerfile=True` and names one prebuilt image per task from `image_template`, default `zenml-bench/{task}:0.1.0`. Local docker runtime: build them with `scripts/build_task_images.sh` in the repo (needs the base image, `docker build -t zenml-bench/base:0.96.4 shared/base`). Prime sandboxes: push the 18 images to a registry and pass `--env.taskset.image-template \"<registry>/{task}:0.1.0\"`.\n\n### Quickstart\n```bash\nuv pip install -e .                      # or: prime env install zenml/zenml-bench\nuv run eval zenml-bench -m gpt-5.4-mini --client.base-url https://api.openai.com/v1 --client.api-key-var OPENAI_API_KEY \\\n  --env.agent.harness.id codex --env.agent.runtime.type docker --no-push\n```\n`eval` is verifiers' v1 runner (`vf-eval` is the legacy one and cannot load a v1 taskset). Outputs: `outputs/<run>/traces.jsonl` + `configs/resolved/eval.json`. The taskset resolves the dataset from the repo's `registry.json` at tag `v0.1` (Harbor's `--repo` route), so it works before the Harbor Hub entry is public. `ignore_timeouts=false` (the default here) keeps the task-authored budgets; the research tasks state their wall-clock budget in the instruction.\n\n### Environment arguments\n| arg | default | description |\n|---|---|---|\n| `dataset` | `zenml-bench@0.1.0` | Harbor dataset id (bare name with `repo`/`registry_path`; `org/name@ref` on the Hub) |\n| `repo` | `zenml-io/zenml-bench@v0.1` | git registry to resolve from; set `None` once the Hub id is used |\n| `image_template` | `zenml-bench/{task}:0.1.0` | prebuilt image per task (`{task}` = task directory name) |\n| `tasks` | all | subset of task names |\n| `timeout_multiplier` | 1.0 | scale agent/verifier timeouts |\n","encoding":"utf-8","truncated":false,"total_bytes":3231},"status":null}