{"data":{"kind":"file","path":"README.md","version_id":"ymxa5b9uh2vvzutod8wtmh76","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":3194,"modified_at":"2026-08-14T10:12:34.355000","content_hash":"821473a00905006cf7477c889c4c86ed23d3c9106f2c3f605233b476a27208f5"},"entries":[],"content":"# shell-ops\n\nVerifiable command-line tasks for LLM agents, scored by an independent verifier\nthe agent never sees.\n\n## What it measures\n\nWhether an agent can drive a Linux shell to a required end state. Each task\ndrops the agent into a fresh sandbox with a `bash` tool, builds a hermetic\nstarting state, lets the agent work, then runs a verifier that inspects the\nfinal filesystem. Reward is binary on the verifier's exit code — there is no\nstring matching and no partial credit.\n\n## Design\n\nBuilt on `verifiers.SandboxEnv` (a `StatefulToolEnv`): every rollout gets its\nown Prime sandbox, the agent interacts through a persistent `bash` tool, and the\nverifier runs at teardown — the agent never sees it. Three properties are\ndeliberate, because they are what separate a usable reward signal from a\ndecorative one:\n\n**Verifiers re-derive the expected result.** `log-error-count` runs\n`grep -c '^ERROR' app.log` itself rather than comparing against a hardcoded\ncount; `json-deep-merge` recomputes the merge and compares parsed JSON, not\ntext. A test that can be satisfied by echoing a constant measures nothing.\n\n**Reward requires constructing the artifact, not guessing a value.**\n`git-restore-deleted` writes a *random* secret into the file at setup, so it\nlives only in git history — fabricating plausible contents fails; the agent must\nactually recover it from the repository, and the verifier reads the expected\ncontents back out of git history to check.\n\n**Starting state is built in-sandbox, not shipped.** Every task constructs its\nown fixtures with a seeded generator or a heredoc, so a rollout never depends on\nnetwork access or a prior task's leftovers. Setup and verifier scripts are\nbase64-encoded into the sandbox so quotes, heredocs and newlines in task content\ncan never break command construction.\n\n## Tasks\n\n| id | tests | non-gameable because |\n|---|---|---|\n| `log-error-count` | Filtering and counting structured log lines | verifier re-counts from the log |\n| `csv-recover-malformed` | Row-level validation while preserving order | verifier re-filters from source, exact match |\n| `git-restore-deleted` | Recovering a file deleted in an earlier commit without rewriting history | contents are a random secret only in git history |\n| `json-deep-merge` | Recursive object merge with override semantics | verifier recomputes the merge, compares parsed JSON |\n\nEach verifier has been checked against correct and adversarial (gaming)\nsolutions in a real Linux container: every correct solution passes and every\nshortcut — wrong constant, verbatim copy, shallow merge, fabricated file — fails.\n\n## Adding tasks\n\nAppend a dict to `TASKS` in `shell_ops.py` with `id`, `prompt`, `setup` and\n`verify`. Nothing else changes. `verify` must exit non-zero on failure and must\nnot trust any value the agent could have printed — re-derive it from the state,\nor randomise it at setup so it cannot be guessed.\n\n## Usage\n\n```bash\nuv pip install -e .\nuv run vf-eval shell-ops\n```\n\nRunning a rollout spins up a Prime sandbox, so `prime login` (an API key) is\nrequired first. The base image defaults to `python:3.11` (ships `git`); override\nwith `load_environment(docker_image=...)`.\n","encoding":"utf-8","truncated":false,"total_bytes":3194},"status":null}