{"data":{"kind":"file","path":"README.md","version_id":"b1xmrc1wt1kxr7aclnby2dop","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":6667,"modified_at":"2026-09-16T14:29:37.189000","content_hash":"2f732dfbc73c61d6bef7d9af66a5f2a3d5876f267716b1df7551a924eafed475"},"entries":[],"content":"# API Debugging Environment\n\nA repository-level coding environment built with Verifiers v1. An LLM agent\nreceives one of eight broken FastAPI projects in Docker, uses terminal and edit\ntools to repair it, and earns a deterministic test-based reward.\n\n## Architecture\n\n```text\neight task definitions + broken repositories\n                    ↓\n          ApiDebuggingTaskset\n                    ↓\n           ApiDebuggingEnv\n                    ↓\n     Docker runtime + Bash harness\n                    ↓\n       model reads/edits/runs pytest\n                    ↓\n     protected tests injected at scoring\n                    ↓\n        all pass: 1.0, else: 0.0\n```\n\n- **Dataset:** eight fixed, reproducible tasks under `tasks/`.\n- **Environment:** `ApiDebuggingEnv` runs a single coding agent.\n- **Harness:** Verifiers' Bash harness exposes command execution and file edits.\n- **Sandbox:** each rollout gets a fresh Docker container with networking off.\n- **Verifier:** hidden pytest tests are copied in only after the agent finishes.\n- **Reward:** all hidden tests pass gives `1.0`; any failure gives `0.0`.\n\nThe agent can see its instruction, application code, and public tests. It cannot\nsee the selected task's `tests/test_hidden.py` during the rollout.\n\n## Included tasks\n\n| Task | Seeded defect |\n| --- | --- |\n| `incorrect-status-code` | Creation needs 201 semantics, `Location`, persistence, and increasing IDs |\n| `missing-authentication` | Two admin routes need one reusable Bearer-authentication policy |\n| `validation-bug` | Numeric ranges and non-finite query values must be rejected declaratively |\n| `pagination-error` | Category filtering must precede one-based pagination and total calculation |\n| `database-transaction` | Transfers must roll back even when a database write fails unexpectedly |\n| `user-not-found` | Missing and suspended users require distinct errors without leaking internal state |\n| `request-schema` | Strict integer, whitespace, range, length, and extra-field rules are missing |\n| `cache-defect` | Updates need per-product invalidation without evicting unrelated cached reads |\n\n## Local setup\n\nThe Prime-compatible Python project lives under the required `proj/` directory.\nRun project commands from there:\n\n```bash\ncd /Users/hridayeshadhikari/workspace-2/environments/api_debugging/proj\nuv venv --python 3.13\nsource .venv/bin/activate\nuv pip install -e '.[dev]'\n./scripts/build_image.sh\n```\n\nThen run the package checks:\n\n```bash\nuv run pytest\nuv run validate api-debugging --runtime.type docker\n```\n\nThe broken image is expected to fail its public missing-user test before an\nagent repairs it. The environment/package tests themselves should pass.\n\n## Evaluate with Ollama Cloud\n\nThe included evaluation config connects directly to Ollama's hosted\nOpenAI-compatible endpoint. It does not use the Ubuntu Ollama server or an SSH\ntunnel.\n\nCreate an API key at <https://ollama.com/settings/keys>. Then, from this\nenvironment directory:\n\n```bash\nsource .venv/bin/activate\nexport OLLAMA_API_KEY=\"your-real-ollama-api-key\"\ncurl https://ollama.com/v1/models \\\n  -H \"Authorization: Bearer ${OLLAMA_API_KEY}\"\nuv run eval @ configs/ollama.toml\n```\n\nThe default model is `gpt-oss:20b`, which completed the starter task with reward\n`1.0`. Override it with `-m MODEL_NAME` when needed. The API key is read only from\n`OLLAMA_API_KEY`; never put it in `configs/ollama.toml` or commit it to Git.\n`eval` is the current Verifiers v1 command; `vf-eval` is the legacy evaluator\nand does not run this v1 task/runtime design.\n\nThe config runs all eight tasks with one rollout per task. To run only a subset,\noverride `--num-tasks`; increase rollouts only after a single pass works reliably.\n\n## Build with Prime/OpenEnv\n\nPrime 0.6.34 requires `environments/<environment_id_underscore>/proj/`. This\nrepository follows that layout and includes an OpenEnv adapter alongside the\noriginal Verifiers environment.\n\nFrom the outer `api_debugging/` directory, build and register the image with:\n\n```bash\nprime env build api-debugging --path ..\n```\n\nThe OpenEnv adapter selects one task per episode. `reset()` returns the bug\ninstruction and visible project files; `step()` accepts a task name and unified\ndiff, applies it to an isolated temporary copy, runs public and protected tests,\nand returns a binary reward. Persistent WebSocket episodes may choose a task\nduring reset; stateless HTTP callers include `task_name` in the step action.\n\n## Folder map\n\n```text\napi_debugging.py                         multi-task loader, environment, reward\nopenenv.yaml                             Prime/OpenEnv server manifest\nmodels.py                                patch action and test-result observations\nserver/app.py                            ASGI/OpenEnv entry point\nserver/api_debugging_environment.py      isolated patch application and scoring\nserver/Dockerfile                        Prime image definition\nconfigs/ollama.toml                      local model/evaluation settings\nscripts/build_image.sh                   builds all eight Docker images\nscripts/run_hidden_tests.sh              shared protected-test runner\ntasks/Dockerfile                         shared reproducible image recipe\ntasks/<task>/instruction.md              prompt shown to the agent\ntasks/<task>/task.toml                   task metadata and resource limits\ntasks/<task>/environment/project/        broken repository copied into its image\ntasks/<task>/tests/test_hidden.py         protected verifier tests\ntasks/<task>/solution/solve.sh            maintainer-only reference solution\ntests/                                   environment definition tests\ndocs/ARCHITECTURE.md                     full structure, runtime flow, verifier design\n```\n\nFor a file-by-file walkthrough of what runs when, the rollout lifecycle, and where the\nprotected tests are injected, see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).\n\n## Reward design and future expansion\n\nThe reward is intentionally binary and judge-free:\n\n```text\nall protected tests pass -> 1.0\notherwise                -> 0.0\n```\n\nFor a later benchmark, a partial score can be calculated as\n`passed_tests / total_tests`, but binary scoring is easier to audit. Every task\nalso has a maintainer solution used by `validate` to prove attainability.\n\n## Publication note\n\nThe tasks currently name local Docker images. Before publishing for remote\nworkers, push all eight images to a registry and replace each `docker_image` in\nits `task.toml` with an immutable registry reference, ideally a digest. The\ndefault image in `api_debugging.py` and `configs/ollama.toml` should also point\nto a remotely available fallback image.\n","encoding":"utf-8","truncated":false,"total_bytes":6667},"status":null}