{"data":{"kind":"file","path":"README.md","version_id":"pg0hvbg3n3tm76qe92bhygrb","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2666,"modified_at":"2026-08-10T14:15:22.903000","content_hash":"11e4475941760db65b434333aa7339051eb0572bdcbfe18fcecfc68ffe139832"},"entries":[],"content":"# FastAPI Agent Debug\r\n\r\nA sandboxed coding-agent evaluation environment for testing an AI agent's\r\nability to diagnose and repair realistic FastAPI backend bugs.\r\n\r\nThe benchmark contains 10 intentionally broken FastAPI applications.\r\nAn agent receives a debugging instruction, edits the application inside an\r\nisolated sandbox, and is evaluated using hidden pytest tests.\r\n\r\n## What This Environment Evaluates\r\n\r\nThe tasks cover common backend engineering problems involving:\r\n\r\n1. HTTP status codes\r\n2. Async / await handling\r\n3. Dependency injection\r\n4. Pydantic request validation\r\n5. HTTP exception handling\r\n6. API-key authentication\r\n7. Path and query parameter validation\r\n8. Response models and sensitive-field filtering\r\n9. Dependency resource lifecycle management\r\n10. Background task scheduling\r\n\r\n## Evaluation Design\r\n\r\nEach task contains:\r\n\r\n- an intentionally broken `app.py`\r\n- a natural-language debugging instruction\r\n- hidden pytest tests used for evaluation\r\n\r\nDuring a rollout:\r\n\r\n1. The broken application is copied into an isolated sandbox.\r\n2. The coding agent receives the debugging instruction.\r\n3. The agent can inspect and modify the application using shell tools.\r\n4. Hidden tests are injected only after the rollout.\r\n5. Pytest evaluates the resulting implementation.\r\n6. The environment assigns a binary reward:\r\n   - `1.0` if all hidden tests pass\r\n   - `0.0` otherwise\r\n\r\nThis prevents the agent from simply reading the grader implementation.\r\n\r\n## Task Set\r\n\r\n| Task | Area | Bug |\r\n|---|---|---|\r\n| task_001 | HTTP semantics | Resource creation returns HTTP 200 instead of 201 |\r\n| task_002 | Async Python | Coroutine is returned without being awaited |\r\n| task_003 | Dependency injection | Dependency function is executed directly |\r\n| task_004 | Validation | Request body accepts invalid product data |\r\n| task_005 | Error handling | Missing resource returns HTTP 200 instead of 404 |\r\n| task_006 | Authentication | Admin endpoint does not enforce API-key dependency |\r\n| task_007 | Parameter validation | Path/query values lack range constraints |\r\n| task_008 | Response security | Internal password hash leaks in API response |\r\n| task_009 | Resource lifecycle | Database session is never cleaned up |\r\n| task_010 | Background work | Notification work runs directly in request handler |\r\n\r\n## Structure\r\n\r\n```text\r\nfastapi_agent_debug/\r\n├── fastapi_agent_debug.py\r\n├── pyproject.toml\r\n├── README.md\r\n└── tasks/\r\n    ├── task_001/\r\n    │   ├── app.py\r\n    │   ├── instruction.md\r\n    │   └── test_app.py\r\n    ├── task_002/\r\n    ├── ...\r\n    └── task_010/","encoding":"utf-8","truncated":false,"total_bytes":2666},"status":null}