{"data":{"kind":"file","path":"README.md","version_id":"py52vz2tgjmmtqcnaml4go8v","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":9301,"modified_at":"2026-09-12T04:20:07.840000","content_hash":"9ee4eadc35311df9517e9979d32372eaa71cce6d87b0e262068768b4c14a4dac"},"entries":[],"content":"# terminal-agentic-v1\n\nTerminal-Bench 2.0 ([tbench.ai](https://www.tbench.ai/)) loaded through Harbor, wired for\nverifiers evaluation and prime-rl training. The package is a thin wrapper over verifiers'\n`HarborTaskset`: it names the Hub package, marks every task as needing a container, and\nexports the harbor env as the default env. Prompts, images, network policy and verifier\nscripts are the benchmark's own.\n\n| | |\n|---|---|\n| **Archetype** | Harbor-backed benchmark port |\n| **Harness** | `terminus-2` (Harbor's reference agent, named in the upstream README), `claude-code`, `codex`, `bash` |\n| **Runtime** | Container required (`NEEDS_CONTAINER = True`) — Docker locally, Prime for scale |\n| **Default env** | `HarborEnv`; identical to the single-agent env for the shared-verifier tasks tb2 ships |\n| **Reward** | `solved` (weight 1.0): the verdict the task's own `tests/test.sh` leaves in `/logs/verifier/` |\n\n## Tasks\n\n89 Terminal-Bench 2.0 tasks, loaded from the Harbor Hub package\n`terminal-bench/terminal-bench-2`. Each task ships its instruction (used as the prompt,\nstripped of surrounding whitespace), a pullable Docker image, a `tests/test.sh` verifier\nand timeouts; `HarborTaskset`\nmaps them onto `TaskData` (`prompt`, `image`, `workdir`, `network_allow`, `timeout`,\n`resources`).\n\nWhat was verified upstream on GitHub (`harbor-framework/terminal-bench-2`, formerly\n`laude-institute/terminal-bench-2`, `main` at `2fd12b8`), across all 89 `task.toml` files:\n\n- every task declares a pullable `docker_image = \"alexgshaw/<task>:20251031\"`;\n- every task sets `allow_internet = true`, which becomes `network_allow = [\"*\"]`;\n- none sets `[verifier].environment_mode`, so every task grades in the agent's own box;\n- agent timeouts range 600–12000 s and verifier timeouts 360–12000 s (48 of 89 at 900 s\n  on each side, 47 on both), all ignored by default (see Design).\n\nThe Hub package itself was not inspected here (the Harbor CLI is not installed — see\nDependencies), so read the above as the state of the GitHub repository, not a guarantee\nabout the Hub snapshot.\n\n**Unpinned.** `dataset` carries no `@ref` because no Hub tag, revision or digest could be\nverified offline: Hub refs are not Git refs, and the GitHub repository has no tags or\nreleases. Provenance that *was* verified: the legacy Harbor registry\n(`laude-institute/harbor`, `registry.json`) pins `terminal-bench@2.0` to all 89 tasks at\ncommit `69671fbaac6d67a7ef0dfec016cc38a64ef7a77c` (2025-10-31) of the tb2 repository, whose\n`task.toml` files use schema 1.0 and predate the `allow_internet` field. To freeze a Hub\nsnapshot, look a ref up with the Harbor CLI and set it yourself — `dataset` is an\nordinary config knob:\n\n```toml\n[env.taskset]\ndataset = \"terminal-bench/terminal-bench-2@<ref>\"\n```\n\n## Reward\n\nOne reward, `solved` (weight 1.0), inherited unchanged from `HarborTask`. After the agent\nphase it stages the task's `tests/` into `/tests` — first deleting\n`/logs/verifier/reward.json` and `/logs/verifier/reward.txt`, so a verdict the agent planted\nis gone before the grader runs — then runs `bash /tests/test.sh` and reads, in order:\n\n1. `/logs/verifier/reward.json`: a finite number, or a non-empty JSON object of numbers. An\n   object with a `reward` key scores `reward` and records every other key as a metric\n   (`{\"reward\": 1, \"tests_passed\": 3}` → reward 1.0, metric `tests_passed = 3.0`); an object\n   without one records each key as a reward of its own. Parsing is strict — a quoted string,\n   `NaN`, an infinity or `{}` counts as absent.\n2. otherwise `/logs/verifier/reward.txt`, parsed as a float (an empty file reads as 0).\n3. otherwise **0.0**.\n\nA verifier that crashes or writes nothing therefore scores exactly like a failed task, and\nnothing on the trace separates the two. The two upstream tb2 verifiers inspected\n(`adaptive-rejection-sampler` and `build-pov-ray` on `main`) run pytest and end in\n`echo 1|0 > /logs/verifier/reward.txt`, so for those the reward is binary; the other 87 were\nnot read.\n\n## Arguments\n\n`[env.taskset]` is a `HarborConfig`; only `dataset`'s default differs from upstream. Defaults\nare verifiers 0.3.1's.\n\n| Argument | Default | Meaning |\n|---|---|---|\n| `dataset` | `\"terminal-bench/terminal-bench-2\"` | Harbor Hub package, `org/name` or `org/name@ref`. Any Harbor-format package works, so the same wiring can run another registry. |\n| `repo` | `None` | Legacy `--repo` registry selector (`org/repo@ref`); with it `dataset` is a bare `name` or `name@version`. |\n| `registry_path` | `None` | Legacy `--registry-path` selector. |\n| `registry_url` | `None` | Legacy `--registry-url` selector for a raw `registry.json`. |\n| `tasks` | `None` | Subset of task names to load (`None` = all 89). |\n| `ignore_timeouts` | `True` | Drop the tasks' declared agent and verifier timeouts (see Design). |\n| `timeout_multiplier` | `1.0` | Scale the tasks' timeouts; only applies with `ignore_timeouts = false`. |\n| `resource_multiplier` | `1.0` | Scale CPU, memory and disk requests (GPUs unchanged). |\n| `require_image` | `False` | Reject a task that declares no environment instead of running it on the runtime's default image. |\n| `ignore_dockerfile` | `False` | Run a Dockerfile-only task on the harness image instead of rejecting it (the Dockerfile is not built). |\n| `ignore_separate_verifier` | `False` | Grade a separate-verifier task in the agent's box anyway. |\n\nThe env is `HarborEnv`, whose config adds `verifier_runtime` (where a separate-verifier\ntask grades; default: derived from the agent's runtime) and `verifier_retries` (default 2).\n\n## Dependencies\n\n- **Python 3.12** — `requires-python = \">=3.12\"`, because Harbor only resolves on 3.12+.\n- **`verifiers>=0.3.1`** — the only declared dependency. The package installs and imports\n  without Harbor.\n- **The Harbor CLI, from `verifiers[harbor]`** — required to load *any* task: `load()` shells\n  out to `harbor download`, so `validate` and `eval` fail before a single task is parsed\n  without it. It is deliberately **not** declared on this package, not even as an extra:\n  the extra pins `harbor==0.21.0`, whose `litellm` dependency requires `openai<3`, so it\n  re-pins `openai` (3.3.1 → 2.54.0 in the workspace this was developed in), `openai-agents`\n  and `websockets`. A uv workspace folds every member extra into the shared lock — declaring\n  the extra here, even behind a `tool.uv.conflicts` fork, was verified to re-pin those three\n  for every environment in the workspace. Install it into a venv where that is acceptable:\n\n  ```bash\n  uv pip install 'verifiers[harbor]>=0.3.1'      # next to this package, Python 3.12\n  uv sync --python 3.12 --extra harbor            # inside a verifiers checkout (its own hint)\n  ```\n- **Docker** (local) or a **Prime** runtime — every tb2 task carries an image and the task\n  class sets `NEEDS_CONTAINER`, so the subprocess runtime is refused.\n- **A model API key** for the chosen harness, exported the way that harness expects (for\n  example `ANTHROPIC_API_KEY` for `claude-code`). No other secrets.\n\n## Design\n\n`ignore_timeouts` stays at verifiers' default of `True`. Timeouts written into a Harbor task\nwere authored against Harbor's own runtime, so enforcing them here scores the speed of\nwhatever inference stack is behind the run as if it were model capability. For a\nhead-to-head against published numbers, set:\n\n```toml\n[env.taskset]\nignore_timeouts = false\ntimeout_multiplier = 2.0\n```\n\n`NEEDS_CONTAINER = True` is set on the task class rather than left to the images: it also\nholds for a registry whose tasks declare no image, which the default `require_image = false`\nwould otherwise let onto the subprocess runtime. `HarborEnv` is exported so that a registry\ndeclaring `[verifier].environment_mode = \"separate\"` grades in a fresh box instead of\nraising.\n\n## Run\n\nModel-free check — needs the Harbor CLI and Docker, and has **not** been run here:\n\n```bash\nuv run validate terminal-agentic-v1 -n 5 --runtime.type docker\n```\n\nAn eval from the CLI:\n\n```bash\nuv run eval terminal-agentic-v1 -n 5 --model <model-id> \\\n  --env.agent.harness.id claude-code --env.agent.runtime.type docker\n```\n\nor from a config file. This is the whole file:\n\n```toml\nmodel = \"anthropic/claude-sonnet-5\"\nnum_tasks = 5\n\n[env.taskset]\nid = \"terminal-agentic-v1\"\nignore_timeouts = true\n\n[env.agent.harness]\nid = \"claude-code\"\n\n[env.agent.runtime]\ntype = \"docker\"\n```\n\n```bash\nuv run eval @ terminal-bench-2.toml --dry-run   # resolves the config; does not load tasks\nuv run eval @ terminal-bench-2.toml\n```\n\n`--dry-run` succeeds even without the Harbor CLI, so it is not evidence that tasks load.\n\n## Status\n\n| Check | Result |\n|---|---|\n| `ruff check`, `ruff format --check` | clean |\n| import | ok — `taskset_class` resolves `TerminalBench2Taskset`, `environment_class` resolves `HarborEnv` |\n| `eval --dry-run` on the config above | resolves |\n| `pytest tests/test_terminal_agentic.py` | 22 passed: loader wiring, the container flag, and the `solved` reward against planted, malformed and missing verdicts on a stub runtime |\n| `validate` | **not run** — the Harbor CLI is not installed (`verifiers[harbor]` re-pins `openai` 3.3.1 → 2.54.0; see Dependencies) |\n| model eval | **not run** — there are no measured reward numbers for this environment |\n","encoding":"utf-8","truncated":false,"total_bytes":9301},"status":null}